Synthetic generation of immunohistochemical special stains

ABSTRACT

There is provided a computer implemented method for training a virtual stainer machine learning model, comprising: creating an imaging multi-record training dataset, wherein a record comprises: a first image of a sample of tissue of a subject stained with a removable stain, and a ground truth indicated by a second image of the sample of tissue of the subject stained with a permanent stain, and training a virtual stainer machine learning model on the imaging multi-record training dataset for generating a virtual image depicting the permanent stain in response to an input image depicting the removable stain.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalApplication No. 63/311,089, filed Feb. 17, 2022, the contents of whichare incorporated herein by reference in its entirety.

FIELD

The present disclosure relates generally to methods, devices, reagents,and kits for use in multiplexed assays for detecting target molecules.Such methods have a wide utility in diagnostic applications, in choosingappropriate therapies for individual patients, or in training neuralnetworks or developing algorithms for use in such diagnosticapplications or selection of therapies. Further, the present disclosureprovides novel compounds for use in detection of target molecules. Thepresent disclosure also relates to methods, systems, and apparatuses forimplementing annotation data collection and autonomous annotation, and,more particularly, to methods, systems, and apparatuses for implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, for feature of interest identification, and/or forvirtual staining of biological samples.

BACKGROUND

Histological specimens are frequently disposed upon glass slides as athin slice of patient tissue fixed to the surface of each of the glassslides. Using a variety of chemical or biochemical processes, one orseveral colored compounds may be used to stain the tissue todifferentiate cellular constituents which can be further evaluatedutilizing microscopy. Brightfield slide scanners are conventionally usedto digitally analyze these slides.

When analyzing tissue samples on a microscope slide, staining the tissueor certain parts of the tissue with a colored or fluorescent dye can aidthe analysis. The ability to visualize or differentially identifymicroscopic structures is frequently enhanced through the use ofhistological stains. Hematoxylin and eosin (H&E) stains are the mostcommonly used stains in light microscopy for histological samples.

In addition to H&E stains, other stains or dyes have been applied toprovide more specific staining and provide a more detailed view oftissue morphology. Immunohistochemistry (“IHC”) stains have greatspecificity, as they use a peroxidase substrate or alkaline phosphatase(“AP”) substrate for IHC stainings, providing a uniform staining patternthat appears to the viewer as a homogeneous color with intracellularresolution of cellular structures, e.g., membrane, cytoplasm, andnucleus. Formalin Fixed Paraffin Embedded (“FFPE”) tissue samples,metaphase spreads or histological smears are typically analyzed bystaining on a glass slide, where a particular biomarker, such as aprotein or nucleic acid of interest, can be stained with H&E and/or witha colored dye, hereafter “chromogen” or “chromogenic moiety.” IHCstaining is a common tool in evaluation of tissue samples for thepresence of specific biomarkers. IHC stains are precise in therecognition of specific targets in throughout the sample and allowquantification of these targets. IHC staining employs chromogenic and/orfluorescent reporters that mark targets in histological samples. This iscarried out by linking the biomarker directly or indirectly with anenzyme, typically either Horse Radish Peroxidase (“HRP”) or AlkalinePhosphatase (“AP”), that subsequently catalyzes the formation of aninsoluble colored or fluorescent precipitate, at the location of thebiomarker from a soluble suitable enzyme substrate, which exhibits acolor or fluorescence.

In situ hybridization (“ISH”) may be used to detect target nucleic acidsin a tissue sample. ISH may employ nucleic acids labeled with a directlydetectable moiety, such as a fluorescent moiety, or an indirectlydetectable moiety, such as a moiety recognized by an antibody which canthen be utilized to generate a detectable signal.

Compared to other detection techniques, such as radioactivity,chemo-luminescence or fluorescence, chromogens generally suffer frommuch lower sensitivity, but have the advantage of a permanent, plainlyvisible color which can be visually observed, such as with bright fieldmicroscopy. However, the advent of new and useful substrates hasalleviated some of these disadvantages. For example, the use ofchromogenic peroxidase substrates has been described in U.S. Pat.10,526,646, the disclosure of which is incorporated herein by referencein its entirety. However, more substrates with additional propertieswhich may be useful in various applications, including multiplexedassays, such as IHC or ISH assays, are needed.

With the advent of advanced deep learning methods, additionalcapabilities for advanced image analysis of histological slides mayimprove detection and assessment of specific molecular markers, tissuefeatures, and organelles, or the like.

A challenge with such analysis methods (which may be used, for example,in digital pathology) is that they require vast amounts of trainingdata. In some cases, training data may require tens of thousands ofmanually identified and annotated objects (e.g., tissue features,organelles, molecular markers, etc.) and can be an extremelytime-consuming process. In addition, even experienced professionals mayencounter difficulties evaluating tissue specimens microscopically toprovide accurate interpretation of patient specimens. For example,detecting lymphocytes within a tumor stained for PD-L1 antigen can bedifficult or impossible to do with an acceptably low error rate. Oneoption is to use serial sections of tissue, where different biologicalmarkers are stained for in each of the two slides, which are thendigitalized and aligned, and the staining pattern on one slide is usedto annotate the other slide. One problem with this approach is that thediscrete digitized layers or sections may be spatially different (e.g.,in terms of orientation, focus, signal strength, and/or the like) fromone another (e.g., with differences among annotation layers and traininglayers, or the like). Consequently, image alignment may be broadlycorrect or accurate for larger or more distinct objects (such as bulktumor detection) but may not have cell-level or organelle-levelprecision, or the like, since the same cells are not necessarily presenton the two slides or images. This can create challenges in developingsuitable training data, which may not be well suited for identifyingsingle cells, organelles, and/or molecular markers, or the like.

SUMMARY

According to a first aspect, a computer implemented method for traininga virtual stainer machine learning model, comprises: creating an imagingmulti-record training dataset, wherein a record comprises: a first imageof a sample of tissue of a subject stained with a removable stain, and aground truth indicated by a second image of the sample of tissue of thesubject stained with a permanent stain, and training a virtual stainermachine learning model on the imaging multi-record training dataset forgenerating a virtual image depicting the permanent stain in response toan input image depicting the removable stain.

In a further implementation form of the first aspect, the removablestain is selected from a group consisting of: Hematoxyline and Eosin(H&E) and a non-labelled scan.

In a further implementation form of the first aspect, the non-labelledscan is selected from a group consisting of: raman spectroscopy,autofluorescence, darkfield, and pure contrast.

In a further implementation form of the first aspect, the permanentstain is selected from a group consisting of: a non-H&E stain, and aspecial stain.

In a further implementation form of the first aspect, the special stainis selected from a group comprising: Masson’s trichrome, Jones SilverH&E, and Periodic Acid - Schiff (PAS).

In a further implementation form of the first aspect, the second imageis created using the sample of tissue depicted in the first image afterthe first image is captured, wherein the sample of tissue in the firstimage is treated to remove the removable stain to create a clearedtissue sample, wherein the permanent stain is applied to the clearedtissue sample to create a permanently stained sample, wherein the secondimage depicts the permanently stained sample.

In a further implementation form of the first aspect, furthercomprising: for the record: identifying a first plurality of biologicalfeatures depicted in the first image, identifying a second plurality ofbiological features depicted in the second image that correspond to theidentified first plurality of biological features depicted in the firstimage, applying an optical flow process and/or non-rigid registrationprocess to the respective second image to compute a respective alignedimage, the optical flow process and/or non-rigid registration processaligns pixels locations of the second image to corresponding pixellocations of the first image using optical flow computed between thesecond plurality of biological features and the first plurality ofbiological features, wherein the ground truth indicated by the secondimage comprises the aligned image.

In a further implementation form of the first aspect, furthercomprising: creating a plurality of imaging multi-record trainingdatasets, each imaging multi-record training dataset comprising adifferent permanent stain depicted in the ground truth indicated by thesecond image, and training a plurality of virtual stainer machinelearning models on the plurality of imaging multi-record trainingdatasets.

In a further implementation form of the first aspect, the virtualstainer machine learning model comprises a generative adversarialnetwork (GAN) comprising a generator network and discriminator network,wherein training comprises training the generative network forgenerating the virtual image using a loss function computed based ondifferences between pixels of the second image of a record and pixels ofan outcome of a virtual image created by the generative network inresponse to an input of the first image of the record, and according toan ability of the discriminator network to differentiate between theoutcome of the virtual image created by the generative network and thesecond image.

According to a second aspect, a computer implemented method forgenerating an image of virtually stained tissue, comprises: feeding atarget image of a sample of tissue of a subject stained with a removablestain into a virtual stainer machine learning model trained according toclaim 1, and obtaining a synthetic image of the sample depicting thesample of tissue stained with a permanent stain.

In a further implementation form of the second aspect, furthercomprising selecting at least one virtual stainer machine learning modelfrom a plurality of virtual stainer machine learning models each trainedon a different imaging multi-record training dataset comprising adifferent permanent stains depicted in the ground truth indicated by thesecond image, wherein the target image is fed into each selected virtualstainer machine learning model.

In a further implementation form of the second aspect, furthercomprising analyzing the target image to identify a type of the sampleof tissue from a plurality of types of sample tissue, and selecting theat least one virtual stainer machine learning model according to thetype of the sample, wherein the plurality of virtual stainer machinelearning models are trained on different imaging multi-record trainingdataset comprising different permanent stains for different types oftissues.

According to a third aspect, a computer implemented method for traininga virtual stainer machine learning model, comprises: creating an imagingmulti-record training dataset, wherein a record comprises: a first imageof a sample of tissue of a subject stained with a permanent stain, aground truth indicated by a second image of the sample of tissue of thesubject stained with a removable stain, and training a virtual stainermachine learning model on the imaging multi-record training dataset forgenerating a virtual image depicting the removable stain in response toan input image depicting the permanent stain.

In a further implementation form of the third aspect, the sample oftissue is first stained with the removable stain to create a removablestain stained sample, the second image of the removable stain stainedsample is captured prior to capture of the first image, the removablestain stained sample is treated to clear the removable stain to create acleared tissue sample, the permanent stain is applied to the clearedtissue sample to create a permanent stain stained sample, wherein thefirst image depicting the permanent stain stained sample is captured.

According to a fourth aspect, a computer implemented method forgenerating an image of virtually stained tissue, comprises: feeding atarget image of a sample of tissue of a subject stained with a permanentstain into a virtual stainer machine learning model trained according toclaim 12, and obtaining a synthetic image of the sample depicting thesample of tissue stained with a H&E stain.

In a further implementation form of the fourth aspect, the permanentstain comprises an immunohistochemistry (IHC) marker stain.

In a further implementation form of the fourth aspect, the permanentstain is selected from a group comprising PDL1, HER2immunohistochemistry (IHC) stain.

According to a fifth aspect, a computer implemented method forgenerating an image of virtually stained tissue, comprises: feeding atarget image of a sample of tissue of a subject stained with a removablestain into a virtual stainer machine learning model running on acomputer, and generating a synthetic image of the sample depicting thesample of tissue stained with a permanent stain as an outcome of thevirtual stainer machine learning model.

According to a sixth aspect, a computing device comprises at least oneprocessor executing a code for at least one of training a virtualstainer machine learning model and performing inference by the virtualstainer machine learning model, according to feature(s) of methodsdescribed herein (e.g., features of the methods described with referenceto the first, second, third, fourth, fifth aspects, and/or otherembodiments described herein).

According to a seventh aspect, a non-transitory medium storing programinstructions for at least one of training a virtual stainer machinelearning model and performing inference by the virtual stainer machinelearning model, which when executed by at least one processor, cause theat least one processor to perform features according to methodsdescribed herein (e.g., features of the methods described with referenceto the first, second, third, fourth, fifth aspects, and/or otherembodiments described herein).

According to an aspect of some embodiments, there is provided a computerimplemented method for training a ground truth generator machinelearning model, the method comprises: creating a ground truthmulti-record training dataset wherein a record comprises: a first imageof a sample of a tissue of a subject depicting a first group ofbiological objects, a second image of the sample of tissue depicting asecond group of biological objects presenting at least one biomarker,and ground truth labels indicating a respective biological objectcategory of a plurality of biological object categories for biologicalobject members of the first group and the second group, and training theground truth generator machine learning model on the ground truthmulti-record training dataset for automatically generating ground truthlabels selected from the plurality of biological object categories forbiological objects depicted in an input set of images of the first typeand the second type.

Optionally, the first image depicts the first group of biologicalobjects presenting at least one first biomarker, and the second imagedepicts the second group of biological objects presenting at least onesecond biomarker different than the at least one first biomarker.

Optionally, the first image comprises a brightfield image, and thesecond images comprises at least one of: (i) a fluorescent image withfluorescent markers indicating the at least one biomarker, (ii) spectralimaging image indicating the at least one biomarker, and (iii)non-labelled image depicting the at least one biomarker.

Optionally, the first image depicts tissue stained with a first staindesigned to stain the first group of biological objects depicting the atleast one first biomarker, wherein the second image depicts the tissuestained with the first stain further stained with a second sequentialstain designed to stain the second group of biological objects depictingat least one second biomarker, wherein the second group includes a firstsub-group of the first group, and wherein a second sub-group of thefirst group is unstained by the second stain and excluded from thesecond group.

Optionally, the method further comprising: feeding unlabeled sets ofimages of samples of tissue of a plurality of sample individuals intothe ground truth generator machine learning model to obtainautomatically generated ground truth labels, wherein the unlabeled setsof images depict biological objects respectively presenting the at leastone first biomarker and at least one second biomarker, and creating asynthetic multi-record training dataset, wherein a synthetic recordcomprises images depicting the at least one first biomarker and excludesimages depicting the at least one second biomarker, labelled with theautomatically generated ground truth labels obtained from the groundtruth generator machine learning model.

Optionally, the method further comprising: training a biological objectmachine learning model on the synthetic multi-record training datasetfor generating an outcome of at least one of the plurality of biologicalobject categories for respective target biological objects depicted in atarget image depicting biological objects presenting at least one firstbiomarker.

Optionally, the method further comprising: training a diagnosis machinelearning model on a biological object category multi-record trainingdataset comprising images depicting biological objects presenting atleast one first biomarker, wherein biological objects depicted in theimages are labelled with at least one of the plurality of biologicalobject categories obtained as an outcome of the biological objectmachine learning model in response to input of the images, whereinimages are labelled with ground truth labels indicative of a diagnosis.

Optionally, ground truth labels of biological objects presenting the atleast one second biomarker depicted in one respective image of arespective set are mapped to corresponding non-labeled biologicalobjects presenting the at least one first biomarker depicted in anotherrespective image of the set, wherein synthetic records of biologicalobjects depicting the at least one first biomarker are labelled withground truth labels mapped from biological objects depicting the atleast one second biomarker.

Optionally, the method further comprising: creating an imagingmulti-record training dataset, wherein a record comprises a first imageof a sample of tissue of a subject depicting a first group of biologicalobjects presenting the at the least one first biomarker, and a groundtruth indicated by a corresponding second image of the sample of tissuedepicting a second group of biological objects presenting at least onesecond biomarker, and training a virtual stainer machine learning modelon the imaging multi-record training dataset for generating a virtualimage depicting biological objects presenting the at least one secondbiomarker in response to an input image depicting biological objectspresenting the at the least one first biomarker.

Optionally, the method further comprising: for each respective record:identifying a first plurality of biological features depicted in thefirst image, identifying a second plurality of biological featuresdepicted in the second image that correspond to the identified firstplurality of biological features, applying an optical flow processand/or non-rigid registration process to the respective second image tocompute a respective aligned image, the optical flow process and/ornon-rigid registration process aligns pixels locations of the secondimage to corresponding pixel locations of the first image using opticalflow computed between the second plurality of biological features andthe first plurality of biological features, wherein the ground truthcomprises the aligned image.

Optionally, the second image of the same sample included in the groundtruth training dataset is obtained as an outcome of the virtual stainermachine learning model fed the corresponding first image.

Optionally, second images fed into the ground truth generator machinelearning model to generate a synthetic multi-record training dataset fortraining a biological object machine learning model, are obtained asoutcomes of the virtual stainer machine learning model fed respectivefirst images.

Optionally, a biological object machine learning model is trained on asynthetic multi-record training dataset including sets of first andsecond images labelled with ground truth labels selected from aplurality of biological object categories, wherein the biological objectmachine learning model is fed a first target image depicting biologicalobjects presenting the at least one first biomarker and a second targetimage depicting biological objects presenting at least one secondbiomarker obtained as an outcome of the virtual stainer machine learningmodel fed the first target image.

Optionally, the first image is captured with a first imaging modalityselected to label the first group of biological objects, wherein thesecond image is captured with a second imaging modality different fromthe first imaging modality selected to visually highlight the secondgroup of biological objects depicting the at least one biomarker withoutlabelling the second group of biological objects.

Optionally, the method further comprising: segmenting biological visualfeatures of the first image using an automated segmentation process toobtain a plurality of segmentations, mapping the plurality ofsegmentation of the first image to the second image, for respectivesegmentations of the second image: computing intensity values of pixelswithin and/or in proximity to a surrounding of the respectivesegmentation indicating visual depiction of at least one secondbiomarker, and classifying the respective segmentation by mapping thecomputed intensity value using a set of rules to a classificationcategory indicating a respective biological object type.

According to another aspect of some embodiments, there is provided acomputer implemented method of automatically generating ground truthlabels for biological objects, the method comprises: feeding unlabeledsets of images of samples of tissue of a plurality of sample individualsinto a ground truth generator machine learning model, wherein theunlabeled sets of images depict biological objects respectivelypresenting at least one first biomarker and at least one secondbiomarker different than the at least one first biomarker, and obtainingautomatically generated ground truth labels as an outcome of the groundtruth generator machine learning model, wherein the ground truthgenerator machine learning model is trained on a ground truthmulti-record training dataset wherein a record comprises: a first imageof a sample of tissue of a subject depicting a first group of biologicalobjects presenting the at least one first biomarker, a second image ofthe sample of tissue depicting a second group of biological objectspresenting the at least one second biomarker different than the at leastone first biomarker, and ground truth labels indicating a respectivebiological object category of a plurality of biological objectcategories for biological object members of the first group and thesecond group.

Optionally, in response to receiving a respective image of a respectivesample of tissue of a respective sample individual depicting the firstgroup of biological objects presenting the at least one first biomarker,the respective image is fed into a virtual stainer machine learningmodel, and obtaining as an outcome of the virtual stainer machinelearning model at least one of: (i) a synthetic image used as a secondimage of the unlabeled sets of images, and (ii) a synthetic image usedas the second image of the sample of the record of the ground truthmulti-record training dataset, wherein the virtual stainer machinelearning model is trained on a virtual imaging multi-record, wherein arecord comprises a first image of a sample of tissue of a subjectdepicting a first group of biological objects presenting the at theleast one first biomarker, and a ground truth indicated by acorresponding second image of the sample of tissue depicting a secondgroup of biological objects presenting the at least one secondbiomarker.

Optionally, the further comprising feeding a target image depicting aplurality of biological objects presenting the at least one firstbiomarker into a biological object machine learning model, and obtainingat least one of the plurality of biological object categories forrespective target biological objects depicted in the target image as anoutcome of the biological object machine learning model, wherein thebiological object machine learning model is trained on a syntheticmulti-record training dataset, wherein a synthetic record comprisesimages depicting the at least one first biomarker and excludes imagesdepicting the at least one second biomarker, labelled with theautomatically generated ground truth labels obtained from the groundtruth generator machine learning model.

According to yet another aspect of some embodiments, there is provided acomputer implemented method for generating virtual images, the methodcomprises: feeding a target image of a sample of tissue of a subjectdepicting a first group of biological objects presenting at least onefirst biomarker into a virtual stainer machine learning model, andobtaining a synthetic image of the sample depicting a second group ofbiological objects presenting at least one second biomarker differentthan the at least one first biomarker as an outcome of the virtualstainer machine learning model, wherein the virtual stainer machinelearning model is trained on a virtual imaging multi-record, wherein arecord comprises a first image of a sample of tissue of a subjectdepicting a first group of biological objects presenting the at theleast one first biomarker, and a ground truth indicated by acorresponding second image of the sample of tissue depicting a secondgroup of biological objects presenting the at least one secondbiomarker.

According to an aspect of some embodiments of the present inventionthere is provided a method for detecting multiple target molecules in abiological sample comprising cells, which comprises:

-   contacting the biological sample with one or more first reagents    which generate a detectable signal in cells comprising a first    target molecule;-   contacting the biological sample with one or more second reagents    which are capable of generating a detectable signal in cells    comprising a second target molecule under conditions in which the    one or more second reagents do not generate a detectable signal;-   detecting the signal generated by the one or more first reagents;-   creating conditions in which the one or more second reagents    generate a signal in cells comprising the second molecule; and-   detecting the signal generated by the one or more second reagents.

According to some of any of the embodiments described herein, the methodfurther comprises:

-   obtaining a first digital image of the signal generated by the one    or more first reagents;-   obtaining a second digital image of the signal generated by the one    or more first reagents and the signal generated by the one or more    second reagents; and-   copying a mask of the second digital image to the first digital    image.

According to some of any of the embodiments described herein, thebiological sample comprises one of a human tissue sample, an animaltissue sample, or a plant tissue sample.

According to some of any of the embodiments described herein, the targetmolecules are indicative of at least one of normal cells, cell type,abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, a marker indicative of or associated with a disease,disorder or health condition, or organ structures.

According to some of any of the embodiments described herein, the targetmolecules are selected from the group consisting of nuclear proteins,cytoplasmic proteins, membrane proteins, nuclear antigens, cytoplasmicantigens, membrane antigens and nucleic acids.

According to exemplary embodiments, the target molecules are nucleicacids.

According to exemplary embodiments described herein, the targetmolecules are polypeptides.

According to some of any of the embodiments described herein, the firstreagents comprise a first primary antibody against a first targetpolypeptide, an HRP coupled polymer that binds to the first primaryantibody and a chromogen which is an HRP substrate.

According to some of these embodiments, the second reagents comprise asecond primary antibody against a second target polypeptide, an HRPcoupled polymer that binds to the second primary antibody, afluorescein-coupled long single chained polymer, an antibody againstFITC that binds to the fluorescein-coupled long single chained polymerand is coupled to HRP and a chromogen which is an HRP substrate.

According to some of any of the embodiments described herein, the methodfurther comprises using a digital image of the signals detected in thebiological sample to train a neural network, in accordance with some ofany of the respective embodiments as described herein.

According to some embodiments, there is provided a neural networktrained using the method as described herein. According to anotheraspect of some of any of the embodiments described herein, there isprovided a method for generating cell-level annotations from abiological sample comprising cells, which comprises:

-   a) exposing the biological sample to a first ligand that recognizes    a first antigen thereby forming a first ligand antigen complex;-   b) exposing the first ligand antigen complex to a first labeling    reagent binding to the first ligand, the first labeling reagent    forming a first detectable reagent, whereby the first detectable    reagent is precipitated around the first antigen and visible in    brightfield;-   c) exposing the biological sample to a second ligand that recognizes    a second antigen thereby forming a second ligand antigen complex;-   d) exposing the second ligand antigen complex to a second labeling    reagent binding to the second ligand, the second labeling reagent    comprising a substrate not visible in brightfield, whereby the    substrate is precipitated around the second antigen;-   e) obtaining a first image of the biological sample in brightfield    to visualize the first chromogen precipitated in the biological    sample;-   f) exposing the biological sample to a third labeling reagent that    recognizes the substrate, thereby forming a third ligand antigen    complex, the third labeling reagent forming a second detectable    reagent, whereby the second detectable reagent is precipitated    around the second antigen;-   g) obtaining a second image of the biological sample in brightfield    with the second detectable reagent precipitated in the biological    sample;-   h) creating a mask from the second image; and-   i) applying the mask to the first image so as to obtain an image of    the biological sample annotated with the second antigen.

According to some of any of the embodiments of this aspect of thepresent invention, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

According to some of any of the embodiments of this aspect of thepresent invention, the target molecules are indicative of at least oneof normal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, prior to step a),applying a target retrieval buffer and protein blocking solution to thebiological sample, whereby the first and second antigens are exposed forthe subsequent steps and endogenous peroxidases are inactivated.

According to some of any of the embodiments of this aspect of thepresent invention, the first and third labeling reagents comprise anenzyme that acts on a detectable reagent substrate to form the first andsecond detectable reagents, respectively.

According to some of any of the embodiments of this aspect of thepresent invention, the first and second antigens are non-nuclearproteins.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step b),denaturing the first ligands to retrieve the first antigens available.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step d),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises following step e),

-   i) removing mounting medium from the slide, and-   ii) rehydrating the biological sample.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises following step f),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

According to exemplary embodiments of this aspect of the presentinvention, the first ligand comprises an anti- lymphocyte-specificantigen antibody (“primary antibody”), the first labeling reagentcomprises a horseradish peroxidase (HRP)-coupled polymer capable ofbinding to the primary antibody, the first detectable reagent comprisesa chromogen comprising an HRP substrate comprising HRP Magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB), the second antigencomprises PD-L1, the second ligand comprises anti-PD-L1 antibodies, thesecond labeling reagent comprises an HRP-coupled polymer capable ofbinding to anti-PD-L1 antibodies, the substrate comprisesfluorescein-coupled long single-chained polymer (fer-4-flu linker), athird labeling reagent comprises an anti-FITC antibody coupled to HRP,and the second detectable reagent comprises a chromogen comprising HRPMagenta or DAB.

According to exemplary embodiments of this aspect of the presentinvention, the first antigen comprises PD-L1, the first ligand comprisesanti-PD-L1 antibodies, the first labeling reagent comprises anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, thesubstrate comprises fluorescein-coupled long single-chained polymer(fer-4-flu linker), the second ligand comprises ananti-lymphocyte-specific antigen antibody (“primary antibody”), thesecond labeling reagent comprises a horseradish peroxidase (HRP)-coupledpolymer capable of binding to the primary antibody, the seconddetectable reagent comprises a chromogen comprising an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), a third labeling reagent comprises an anti-FITC antibody coupledto HRP, and the second detectable reagent comprises a chromogencomprising HRP Magenta or DAB.

According to exemplary embodiments of this aspect of the presentinvention, the first labeling reagent comprises a horseradish peroxidase(HRP)-coupled polymer capable of binding to the primary antibody, thefirst detectable reagent comprises a chromogen comprising an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises p40, the secondligand comprises anti-p40 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-p40antibodies, the substrate comprises fluorescein-coupled longsingle-chained polymer (fer-4- flu linker), the third labeling reagentis an anti-FITC antibody coupled to HRP, and the second detectablereagent comprises a chromogen comprising HRP Magenta or DAB.

According to exemplary embodiments of this aspect of the presentinvention, the first antigen comprises p40, the first ligand comprisesanti-p40 antibodies, the first labeling reagent comprises an HRP-coupledpolymer capable of binding to anti-p40 antibodies, the substratecomprises fluorescein-coupled long single-chained polymer (fer-4- flulinker), the second labeling reagent comprises a horseradish peroxidase(HRP)-coupled polymer capable of binding to the primary antibody, thesecond detectable reagent comprises a chromogen comprising an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), a third labeling reagent an anti-FITC antibodycoupled to HRP, and the second detectable reagent comprises a chromogencomprising HRP Magenta or DAB.

According to some of these embodiments, a counterstaining agent ishematoxylin.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises using a digital image ofthe signals detected in the biological sample to train a neural network,as described herein in any of the respective embodiments, and accordingto some embodiments, there is provided a neural network trained usingthis method.

According to some embodiments, there is provided an annotated imageobtained by the method as described herein in any of the respectiveembodiments and any combination thereof.

According to yet another aspect of some embodiments of the presentinvention there is provided a method for detecting multiple targetmolecules in a biological sample comprising cells, which comprises:contacting the biological sample with one or more first reagents whichgenerate a detectable signal in cells comprising a first targetmolecule; contacting the biological sample with one or more secondreagents which generate a detectable signal in cells comprising a secondtarget molecule, wherein the detectable signal generated by the one ormore second reagents is removable; detecting the signal generated by theone or more first reagents and the signal generated by the one or moresecond reagents; creating conditions in which the signal generated bythe one or more second reagents is removed; and detecting the signalgenerated by the one or more first reagents.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises: obtaining a firstdigital image of the signal generated by the one or more first reagentsand the signal generated by the one or more second reagents; obtaining asecond digital image of the signal generated by the one or more firstreagents; and copying a mask of the first digital image to the seconddigital image.

According to some of any of the embodiments of this aspect of thepresent invention, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

According to some of any of the embodiments of this aspect of thepresent invention, the target molecules are indicative of at least oneof normal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

According to some of any of the embodiments of this aspect of thepresent invention, the target molecules are selected from the groupconsisting of nuclear proteins, cytoplasmic proteins, membrane proteins,nuclear antigens, cytoplasmic antigens, membrane antigens and nucleicacids.

According to exemplary embodiments, the target molecules are nucleicacids.

According to exemplary embodiments, the target molecules arepolypeptides.

According to some of any of the embodiments of this aspect of thepresent invention, the first reagents comprise a first primary antibodyagainst a first target polypeptide, an HRP coupled polymer that binds tothe first primary antibody and a chromogen which is an HRP substrate.

According to some of any of the embodiments of this aspect of thepresent invention, the second reagents comprise a second primaryantibody against a second target polypeptide, an HRP coupled polymerthat binds to the second primary antibody, and amino ethyl carbazole.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises using a digital image ofthe signals detected in the biological sample to train a neural network,as described herein in any of the respectrive embodiments, and accordingto some embodiments, there is provided a neural network trained usingthe method.

According to still another aspect of some embodiments of the presentinventuion there is provided a method for generating cell-levelannotations from a biological sample comprising cells, which comprises:

-   a) exposing the biological sample to a first ligand that recognizes    a first antigen thereby forming a first ligand antigen complex;-   b) exposing the first ligand antigen complex to a first labeling    reagent binding to the first ligand, the first labeling reagent    forming a first detectable reagent, whereby the first detectable    reagent is precipitated around the first antigen;-   c) exposing the biological sample to a second ligand that recognizes    a second antigen thereby forming a second ligand antigen complex;-   d) exposing the second ligand antigen complex to a second labeling    reagent binding to the second ligand, the second labeling reagent    forming a second detectable reagent, whereby the second detectable    reagent is precipitated around the second antigen;-   e) obtaining a first image of the biological sample with the first    and second detectable reagents precipitated in the biological    sample;-   f) incubating the tissue sample with an agent which dissolves the    second detectable reagent;-   g) obtaining a second image of the biological sample with the first    detectable reagent precipitated in the biological sample;-   h) creating a mask from the first image; and-   i) applying the mask to the second image so as to obtain an    annotated image of the biological sample with the second antigen.

According to some of any of the embodiments of this aspect of thepresent invention, the first biological sample comprises one of a humantissue sample, an animal tissue sample, or a plant tissue sample,wherein the objects of interest comprise at least one of normal cells,abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, or organ structures.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, prior to step a),applying a target retrieval buffer and protein blocking solution to thebiological sample, whereby the first and second antigens are exposed forthe subsequent steps and endogenous peroxidases are inactivated.

According to some of any of the embodiments of this aspect of thepresent invention, the first and second labeling reagents comprise anenzyme that acts on a detectable reagent substrate to form the first andsecond detectable reagents, respectively.

According to some of any of the embodiments of this aspect of thepresent invention, the first and second antigens are non-nuclearproteins.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step b),denaturing the first ligands to retrieve the first antigens available.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step d),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step e),

-   i) removing mounting medium from the slide, and-   ii) rehydrating the biological sample.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step f),dehydrating and mounting the sample on a slide.

According to some of any of the embodiments of this aspect of thepresent invention, the first antigen comprises a lymphocyte-specificantigen, the first ligand comprises an anti-lymphocyte-specific antigenantibody (“primary antibody”), the first labeling reagent comprises ahorseradish peroxidase (HRP)-coupled polymer capable of binding to theprimary antibody, the first detectable reagent comprises an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises PD-L1, the secondligand comprises anti-PD-L1 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-PD-L1antibodies, the second detectable reagent comprises amino ethylcarbazole (AEC), and the agent which dissolves the second detectablereagent is alcohol or acetone.

According to some of any of the embodiments of this aspect of thepresent invention, the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises p40, the second ligand comprisesanti-p40 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-p40 antibodies, thesecond detectable reagent comprises amino ethyl carbazole (AEC), and theagent which dissolves the second detectable reagent is alcohol oracetone.

According to some of any of the embodiments of this aspect of thepresent invention, a counterstaining agent is hematoxylin.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises using a digital image ofthe signals detected in the biological sample to train a neural network,as described herein in any of the respective embodiments, and accordingto some embodiments, there is provided a neural network trained usingthe method.

According to some embodiments, there is provided an annotated imageobtained by the method as described herein.

According to another aspect of some embodiments of the present inventionthere is provided a method for detecting multiple target molecules in abiological sample comprising cells, which comprises: contacting thebiological sample with one or more first reagents which generate a firstdetectable signal in cells comprising a first target molecule, whereinthe first detectable signal is detectable using a first detectionmethod; contacting the biological sample with one or more secondreagents which generate a second detectable signal in cells comprising asecond target molecule, wherein the second detectable signal isdetectable using a second detection method and is substantiallyundetectable using the first detection method and wherein the firstdetectable signal is substantially undetectable using the seconddetection method; detecting the signal generated by the one or morefirst reagents using the first detection method; and detecting thesignal generated by the one or more second reagents using the seconddetection method.

According to some of any of the embodiments of this aspect of thepresent invention, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

According to some of any of the embodiments of this aspect of thepresent invention, the target molecules are indicative of at least oneof normal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

According to some of any of the embodiments of this aspect of thepresent invention, the target molecules are selected from the groupconsisting of nuclear proteins, cytoplasmic proteins, membrane proteins,nuclear antigens, cytoplasmic antigens, membrane antigens and nucleicacids.

According to exemplary embodiments, the target molecules are nucleicacids.

According to exemplary embodiments, the target moleculaed arepolypeptides.

According to some of any of the embodiments of this aspect of thepresent invention, the first reagents comprise a first primary antibodyagainst a first target polypeptide, an HRP coupled polymer that binds tothe first primary antibody and a chromogen which is an HRP substrate.

According to some of any of the embodiments of this aspect of thepresent invention, the second reagents comprise a second primaryantibody against a second target polypeptide, an HRP coupled polymerthat binds to the second primary antibody, and a rhodamine basedfluorescent compound coupled to a long single chained polymer.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises using a digital image ofthe signals detected in the biological sample to train a neural network,as described herein in any of the respective embodiments, and accordingto some embodiments, there is provided a neural network trained usingthis method.

According to yet another aspect of some embodiments of the presentinvention there is provided a method for generating cell-levelannotations from a biological sample comprising cells, which comprises:

-   a) exposing the biological sample to a first ligand that recognizes    a first antigen thereby forming a first ligand antigen complex;-   b) exposing the first ligand antigen complex to a first labeling    reagent binding to the first ligand, the first labeling reagent    forming a first detectable reagent, wherein the first detectable    reagent is visible in brightfield; whereby the first detectable    reagent is precipitated around the first antigen;-   c) exposing the biological sample to a second ligand that recognizes    a second antigen thereby forming a second ligand antigen complex;-   d) exposing the second ligand antigen complex to a second labeling    reagent binding to the second ligand, the second labeling reagent    forming a second detectable reagent, wherein the second detectable    reagent is visible in fluorescence; whereby the second detectable    reagent is precipitated around the second antigen;-   e) obtaining a first brightfield image of the biological sample with    the first detectable reagent precipitated in the biological sample;-   f) obtaining a second fluorescent image of the biological sample    with the second detectable reagent precipitated in the biological    sample;-   g) creating a mask from the second image; and-   h) applying the mask to the first image so as to obtain an annotated    image of the tissue sample with the second marker.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, prior to step a),applying a target retrieval buffer and protein blocking solution to thebiological sample, whereby the first and second antigens are exposed forthe subsequent steps and endogenous peroxidases are inactivated.

According to some of any of the embodiments of this aspect of thepresent invention, the first biological sample comprises one of a humantissue sample, an animal tissue sample, or a plant tissue sample.

According to some of any of the embodiments of this aspect of thepresent invention, the target molecules are indicative of at least oneof normal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

According to some of any of the embodiments of this aspect of thepresent invention, the first and second labeling reagents comprise anenzyme that acts on a detectable reagent substrate to form the first andsecond detectable reagents, respectively.

According to some of any of the embodiments of this aspect of thepresent invention, the first and second antigens are non-nuclearproteins.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step b),denaturing the first ligands to retrieve the first antigens available.

According to some of any of the embodiments of this aspect of thepresent invention, the method further comprises, following step d),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

According to exemplary embodiments, the first labeling reagent comprisesa horseradish peroxidase (HRP)-coupled polymer capable of binding to theprimary antibody, the first detectable reagent comprises an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises PD-L1, the secondligand comprises anti-PD-L1 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-PD-L1antibodies, and the second detectable reagent comprises arhodamine-based fluorescent compound coupled to a long single-chainpolymer.

According to exemplary embodiments, the first antigen comprises PD-L1,the first ligand comprises anti-PD-L1 antibodies, the first labelingreagent comprises a horseradish peroxidase (HRP)-coupled polymer capableof binding to anti-PD-L1 antibodies, the first detectable reagentcomprises a rhodamine-based fluorescent compound coupled to a longsingle-chain polymer, the second labeling reagent comprises anHRP-coupled polymer capable of binding to the primary antibody, and thesecond detectable reagent comprises an HRP substrate comprising HRPMagenta or 3,3′-diaminobenzidine tetrahydrochloride (DAB).

According to exemplary embodiments, the first antigen comprises alymphocyte-specific antigen, the first ligand comprises ananti-lymphocyte-specific antigen antibody (“primary antibody”), thefirst labeling reagent comprises a horseradish peroxidase (HRP)-coupledpolymer capable of binding to the primary antibody, the first detectablereagent comprises an HRP substrate comprising HRP Magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB), the second antigencomprises p40, the second ligand comprises anti-p40 antibodies, thesecond labeling reagent comprises an HRP-coupled polymer capable ofbinding to anti-p40 antibodies, and the second detectable reagentcomprises rhodamine-based fluorescent compound coupled to a longsingle-chain polymer.

According to some of any of the embodiments of this aspect of thepresent invention, the first antigen comprises p40, the first ligandcomprises anti-p40 antibodies, the first labeling reagent comprises ahorseradish peroxidase (HRP)-coupled polymer capable of binding toanti-p40 antibodies, the first detectable reagent comprisesrhodamine-based fluorescent compound coupled to a long single-chainpolymer, the second antigen comprises a lymphocyte-specific antigen, thesecond ligand comprises an anti-lymphocyte-specific antigen antibody(“primary antibody”), the second labeling reagent comprises HRP-coupledpolymer capable of binding to the primary antibody, and the seconddetectable reagent comprises an HRP substrate comprising HRP Magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB).

According to some of any of the embodiments described herein, acounterstaining agent is hematoxylin.

According to some of any of the embodiments of this aspect of thepresent invention, there is provided an annotated image obtained by thedescribed method.

According to an aspect of some embodiments of the present invention,there is provided a compound of Formula I (CM):

wherein:

-   X is —COORx, —CH2COORx, —CONRxRxx, —CH₂CONRxRxx, which includes the    corresponding spirolactones or spirolactams of Formula I, where the    spiro-ring is formed between X and the carbon on the middle ring of    the tricyclic structure. wherein:    -   Y is ═O, ═NRy, or ═N+RyRyy;    -   Z is O or NRz,-   Wherein:    -   R¹, R², R³, R⁴, R⁵, R⁶, R⁷, R⁸, R⁹, R¹⁰, R^(x), R^(xx), R^(z),        R^(y), and R^(yy) are each independently selected from hydrogen        and a substituent having less than 40 atoms;    -   L is a linker comprising a linear chain of 5 to 29 consecutively        connected atoms; and PS is a peroxidase substrate moiety.

According to some of any of the embodiments that relate to this aspectof the present invention, R¹ is selected from hydrogen, R¹¹, (C1-C20)alkyl or heteroalkyl optionally substituted with one or more of the sameor different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionallysubstituted with one or more of the same or different R¹³ or suitableR¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl optionallysubstituted with one or more of the same or different R¹³ or suitableR¹⁴ groups, or alternatively, R¹ may be taken together with R² to formpart of a benzo, naptho or polycyclic aryleno group which is optionallysubstituted with one or more of the same or different R¹³ or suitableR¹⁴ groups;

-   R² is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or    alternatively, R² may be taken together with R¹, to form part of a    benzo, naptho or polycyclic aryleno group which is optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups;-   R^(X), when present, is selected from hydrogen, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups;-   R^(xx), when present, is selected from (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups;-   R³ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups;-   R⁴ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively, when Y is —N⁺R^(Y)R^(YY), R⁴ may be taken together    with Ryy to form a 5- or 6-membered ring which is optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups;-   R^(yy), when present, is selected from (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively R^(yy) may be taken together with R⁴ to form a 5- or    6-membered ring which is optionally substituted with one or more of    the same or different R¹³ or suitable R¹⁴ groups;-   R^(y), when present, is selected from hydrogen, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R14 groups, or, alternatively, R^(Y) may be taken    together with R⁵ to form a 5- or 6-membered ring optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups;-   R^(z), when present, is selected from hydrogen, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups;-   R⁵ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively, R⁵ may be taken together with R⁶ to form part of a    benzo, naptho or polycyclic aryleno group which is optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups, or alternatively, when Y is —N⁺R^(Y)R^(Y)Y, R⁵    may be taken together with Ry to form a 5- or 6-membered ring    optionally substituted with one or more of the sameor different R¹³    or suitable R¹⁴ groups;-   R⁶ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively, R⁶ together with R⁵ may form part of a benzo, naptho    or polycyclic aryleno group which is optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups;-   R⁷, R⁸ and R⁹ are each, independently of one another, selected from    hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl optionally substituted    with one or more of the same or different R¹⁴ groups, (C5-C20) aryl    or heteroaryl optionally substituted with one or more of the same or    different R¹³ or suitable R¹⁴ groups and (C6-C40) arylalkyl or    heteroaryl alkyl optionally substituted with one or more of the same    or different R¹³ or suitable R¹⁴ groups;-   R¹⁰ is selected from selected from hydrogen, R¹¹, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups, halo, haloalkyl, —OR¹², —SR¹², —SOR¹²,    —SO₂R¹², and nitrile;-   R¹¹ is selected from —NR¹⁵R¹⁵, —OR¹⁶, —SR¹⁶, halo, haloalkyl, —CN,    —NC, —OCN, —SCN, —NO, —NO₂, —N₃, —S(O)R¹⁶, —S(O)₂R¹⁶, —S(O)₂OR¹⁶,    —S(O)NR¹⁵R¹⁵, —S(O)₂NR¹⁵R¹⁵,—OS(O)R¹⁶, —OS(O)₂R¹⁶, —OS(O)₂NR¹⁵R¹⁵,    —OP(O)₂R¹6, —OP(O)₃R¹⁶R¹⁶, —P(O)₃R¹⁶R¹⁶, —C(O)R¹⁶, —C(O)OR¹⁶,    —C(O)NR¹⁵R¹⁵, —C(NH)NR¹⁵R¹⁵, —OC(O)R¹⁶, —OC(O)OR¹⁶, —OC(O)NR¹⁵R¹⁵    and —OC(NH)NR¹⁵R¹⁵;-   R¹² is selected from (C1-C20) alkyls or heteroalkyls optionally    substituted with lipophilic substituents, (C5-C20) aryls or    heteroaryls optionally substituted with lipophilic substituents and    (C2-C26) arylalkyl or heteroarylalkyls optionally substituted with    lipophilic substituents;-   R¹³ is selected from hydrogen, (C1-C8) alkyl or heteroalkyl,    (C5-C20) aryl or heteroaryl and (C6-C28) arylalkyl or    heteroarylalkyl;-   R¹⁴ is selected from —NR¹⁵R¹⁵, ═O, —OR¹⁶, ═S, -SR¹⁶, ═NR¹⁶, ═NOR¹⁶,    halo, haloalkyl, —CN, —NC, —OCN, —SCN, —NO, —NO₂, ═N₂, —N₃,    —S(O)R¹⁶, —S(O)₂R¹⁶, —S(O)₂OR¹⁶, —S(O)NR¹⁵R¹⁵, —S(O)₂NR¹⁵R¹⁵,    —OS(O)R¹⁶, —OS(O)₂R¹⁶, —OS(O)₂NR¹⁵R¹⁵,—OS(O)₂OR¹⁶, —OS(O)₂NR¹⁵R¹⁵,    —C(O)R¹⁶, —C(O)OR¹⁶, —C(O)NR¹⁵R¹⁵, —C(NH)NR¹⁵R¹⁵, —OC(O)R¹⁶,    —OC(O)OR¹⁶, —OC(O)NR¹⁵R¹⁵ and —OC(NH)NR¹⁵R¹⁵; each R¹⁵ is    independently hydrogen or R¹⁶, or alternatively, each R¹⁵ is taken    together with the nitrogen atom to which it is bonded to form a 5-    to 8- membered saturated or unsaturated ring which may optionally    include one or more of the same or different additional heteroatoms    and which may optionally be substituted with one or more of the same    or different R¹³ or R¹⁶ groups; each R¹⁶ is independently R¹³ or R¹³    substituted with one or more of the same or different R¹³ or R¹⁷    groups; and each R¹⁷ is selected from —NR¹³R¹³, —OR¹³, ═S, —SR¹³,    ═NR¹³, ═NOR¹³, halo, haloalkyl, —CN, —NC, —OCN, —SCN, —NO, —NO₂,    ═N₂, —N₃, —S(O)R¹³, —S(O)₂R¹³, —S(O)₂OR¹³,—S(O)NR¹³R¹³,    —S(O)₂NR¹³R¹³, —OS(O)R¹³, —OS(O)₂R¹³, —OS(O)₂NR¹³R¹³,    —OS(O)₂OR¹⁶,—OS(O)₂NR¹³R¹³, —C(O)R¹³, —C(O)OR¹³, —C(O)NR¹³R¹³,    —C(NH)NR¹⁵R¹³, —OC(O)R¹³,—OC(O)OR¹³, —OC(O)NR¹³R¹³ and    —OC(NH)NR¹³R¹³.

According to some of any of the embodiments of this aspect of thepresent invention, the compound comprises a chromogenic moiety selectedfrom the group consisting of rhodamine, rhodamine derivatives,fluorescein, fluorescein derivatives in which X is —COOH, —CH₂COOH,—CONH₂, —CH₂CONH₂ and salts of the foregoing.

According to some of any of the embodiments of this aspect of thepresent invention, the compound comprises a chromogenic moiety selectedfrom the group consisting of rhodamine, rhodamine 6G,tetramethylrhodamine, rhodamine B, rhodamine 101, rhodamine 110,fluorescein, O— carboxymethyl fluorescein, derivatives of the foregoingin which X is —COOH, —CH₂COOH, —CONH₂, —CH₂CONH₂ and salts of theforegoing.

According to some of any of the embodiments of this aspect of thepresent invention, the peroxidase substrate moiety has the followingformula:

wherein:

-   R²¹ is —H,-   R²² is —H, —O—Y, or —N(Y)₂; R²³ is —OH or NH₂;-   R²⁴ is —H, —O—Y, or —N(Y)₂;-   R²⁵ is —H, —O—Y, or —N(Y)₂;-   R²⁶ is CO;-   Y is H, alkyl or aryl; wherein PS is linked to L through R²⁶.

According to some of any of the embodiments of this aspect of thepresent invention, R²³ is —OH, and R²⁴ is —H.

According to some of any of the embodiments of this aspect of thepresent invention, either R²¹ or R²⁵ is —OH, R²² and R²4 are —H, and R²³is —OH.

According to some of any of the embodiments of this aspect of thepresent invention, the peroxidase substrate moiety is a residue offerulic acid, cinnamic acid, caffeic acid, sinapinic acid,2,4-dihydroxycinnamic acid or 4-hydroxycinnamic acid (coumaric acid).

According to some of any of the embodiments of this aspect of thepresent invention, the peroxidase substrate moiety is a residue of4-hydroxycinnamic acid.

According to some of any of the embodiments of this aspect of thepresent invention, the linker is a compound (R³⁵) that comprises:

-   wherein the curved lines denote attachment points to the    compound (CM) and to the peroxidase substrate (PS), and wherein R³⁴    is optional and can be omitted or used as an extension of linker,    wherein R³⁴ is:

-   

-   wherein the curved lines denote attachment point to R³³ and PS,    wherein R³¹ is selected from methyl, ethyl, propyl, OCH₂, CH₂OCH₂,    (CH₂OCH₂)₂, NHCH₂, NH(CH₂)₂, CH₂NHCH₂, cycloalkyl, alkyl-cycloalkyl,    alkyl-cycloalkyl-alkyl, heterocyclyl (such as nitrogen-containing    rings of 4 to 8 atoms), alkyl-heterocyclyl,    alkyl-heterocyclyl-alkyl, and wherein no more than three    consecutively repeating ethyloxy groups, and

-   R³² and R³³ are independently in each formula is selected from NH    and O.

According to some of any of the embodiments of this aspect of thepresent invention, the linker is selected from one or two repeat of amoiety of Formula IIIa, IIIb, or IIIc:

with an optional extansion:

which would be inserted between the respective atom N in the compound(CM) and PS bond.

According to some of any of the embodiments of this aspect of thepresent invention, Z-L-PS together comprises:

wherein the curved line denotes the attachment point.

According to some of any of the embodiments of this aspect of thepresent invention, the compound has the formula:

According to some of any of the embodiments of this aspect of thepresent invention, the compound has a formula selected from the groupconsisting of:

and their corresponding spiro-derivates, or salts thereof.

According to some of any of the embodiments of this aspect of thepresent invention, the compound has the formula:

STATEMENT REGARDING COLOR DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the nessecary fee.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

A further understanding of the nature and advantages of particularembodiments may be realized by reference to the remaining portions ofthe specification and the drawings, in which like reference numerals areused to refer to similar components. In some instances, a sub-label isassociated with a reference numeral to denote one of multiple similarcomponents. When reference is made to a reference numeral withoutspecification to an existing sub-label, it is intended to refer to allsuch multiple similar components.

FIGS. 1A and 1B illustrate PD-L1 sequential staining, in accordance withvarious embodiments.

FIGS. 2A and 2B illustrate p40 sequential staining, in accordance withvarious embodiments.

FIG. 3 is a schematic diagram illustrating a system for implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, for features of interest identification, and/or forvirtual staining of biological samples, in accordance with variousembodiments.

FIGS. 4A-4C are process flow diagrams illustrating various non-limitingexamples of training of one or more artificial intelligence (“AI”)models and inferencing performed by a trained AI model when implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, and/or for features of interest identification ofbiological samples, in accordance with various embodiments.

FIGS. 5A-5N are schematic diagrams illustrating a non-limiting exampleof various process steps performed by the system when implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, and/or for features of interest identification ofbiological samples in conjunction with corresponding images of thebiological sample during each process (and in some cases, at varyinglevels of magnification), in accordance with various embodiments.

FIGS. 6A-6E are flow diagrams illustrating a method for implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, and/or for features of interest identification ofbiological samples, in accordance with various embodiments.

FIGS. 7A-7D are process flow diagrams illustrating various non-limitingexamples of training of an artificial intelligence (“AI”) model togenerate a virtual stain and inferencing performed by a trained AI modelwhen implementing sequential imaging of biological samples forgenerating training data for developing deep learning based models forvirtual staining of biological samples and for generating training datafor developing deep learning based models for image analysis, cellclassification, and/or features of interest identification of biologicalsamples, in accordance with various embodiments.

FIGS. 8A-8O are schematic diagrams illustrating various non-limitingexamples of images of biological samples that have been virtuallystained when implementing sequential imaging of biological samples forgenerating training data for developing deep learning based models forvirtual staining of biological samples and for developing deep learningbased models for image analysis, cell classification, and/or features ofinterest identification of biological samples, in accordance withvarious embodiments.

FIGS. 9A-9E are flow diagrams illustrating a method for implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for virtual staining ofbiological samples and for developing deep learning based models forimage analysis, cell classification, and/or features of interestidentification of biological samples, in accordance with variousembodiments.

FIG. 10 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments.

FIG. 11 is a block diagram illustrating a networked system of computers,computing systems, or system hardware architecture, which can be used inaccordance with various embodiments.

FIG. 12 is a block diagram of components of a system for training MLmodels that analyze first and second images of a sample of tissue and/orfor inference of first and second images of a sample of tissue using theML models, in accordance with various embodiments.

FIG. 13 is a flowchart of a method of automatically generating a virtualstainer machine learning model (also referred to herein as virtualstainer) that generates an outcome of a second image in response to aninput of a first image, in accordance with various embodiments.

FIG. 14 is a flowchart of a method of automatically generating anannotated dataset of an image of a sample of tissue, in accordance withvarious embodiments.

FIG. 15 is a flowchart of a method of training a biological objectmachine learning model and/or a diagnosis machine learning model usingunlabeled images of samples of tissue, in accordance with variousembodiments.

FIG. 16 is a flowchart of an alternative deterministic based approachfor automatically annotating biological objects in images of samples oftissue, in accordance with various embodiments.

FIG. 17 is a flowchart of a method of obtaining a diagnosis for a sampleof tissue of a subject, in accordance with various embodiments.

FIGS. 18A-18C are schematics of exemplary architectures of the virtualstainer machine learning model (sometimes referred to herein as “ModelF”), in accordance with various embodiments.

FIG. 19 is a schematic illustration of an exemplary sequential stainingusing a dark quenche, in accordance with various embodiments.

FIG. 20 presents images of a tissue section where multiple round ofstaining has been done using DAB as a quencher between reactions. FirstPDL1 (left), then CD8 for cytotoxic T-cells (center) which wassubsequently dark quenched and stained for B-cells (right).

FIG. 21 presents an immunofluorescence image of an antibody againstLaminB1, and a nuclear envelope protein which effectively stains thenuclear periphery.

FIG. 22 presents the chemical structure of a fer-4-flu linker, asdescribed in U.S. Pat. Application Publication No. 2016/0122800.

FIG. 23 is a flowchart of a method of training a virtual stainer MLmodel, in accordance with various embodiments;

FIG. 24 is a flowchart of a method of inference using a trained virtualstainer ML model, in accordance with various embodiments;

FIG. 25 is a flowchart of an exemplary process for generating a firstimage denoting a training input image, and a ground truth second imageof a record of a multi-record training dataset for training a virtualstainer, in accordance with various embodiments;

FIG. 26 is a dataflow diagram depicting exemplary dataflow for traininga generative adversarial network (GAN) implementation of the virtualstainer, in accordance with various embodiments;

FIG. 27 includes example images of real and virtual slides, as part ofan experiment performed by Inventors, in accordance with variousembodiments;

FIG. 28 includes example images of additional real and virtual slides,as part of another experiment performed by Inventors, in accordance withvarious embodiments;

FIG. 29 includes example images of yet additional real and virtualslides, as part of yet another experiment performed by Inventors, inaccordance with various embodiments; and

FIG. 30 includes example images of yet additional real and virtualslides, as part of yet another experiment performed by Inventors, inaccordance with various embodiments.

DETAILED DESCRIPTION

As used herein, the term removable stain refers to a non-permanent stainthat may be removed. The removable stain may be a physical dye, forexample, Hematoxylin and Eosin (H&E) stain, which may be removed, forexample, as described herein. Alternatively or additionally, theremovable stain may be created from a non-labelled scan, such as byshining electromagnetic energy at the sample of tissue, for example,raman spectroscopy, autofluorescence, darkfield, and pure contrast. Thenon-labelled “stain” may be removed by terminating the electromagneticenergy, for example, stopping the light creating the darkfield image,stopping the fluorescence energy creating the autofluorescence, and thelike.

As used herein, the term permanent stain refers to a non-removable stainthat cannot be removed without causing damage to the tissue sample suchthat pathological data and/or clinical data cannot be reliable obtained.Even if the permanent stain is removed using certain chemicals, thecertain chemicals damage the tissue sample during the removal of thepermanent stain, such that the tissue sample can no longer be used toobtain the desired pathological and/or clinical data.

As used herein, the term special stain may sometimes be interchangedwith the term non-H&E stain and/or may be interchanged with the termpermanent stain, i.e., the term special stain and/or permanent stain maysometimes refer to any stain that is not H&E. A special stain is ahistochemical stain which uses dyes or chemicals with affinity forparticular tissue components or structures and can be used to visualizesaid components or structures using, for example, light or fluorescentmicroscopy techniques. There are many special stains (e.g., hundreds).The specific special stain may be selected according to, for example,the type of tissue being analyzed. For example, some infectiousmicro-organisms such as fungi or bacteria are almost invisible whenexamined with hematoxylin and eosin. When a special stain is added tothe tissue, these micro-organisms turn black or red which makes themmuch easier to see. Special stains include histochemical stains whichare primarily used for more simple enhancements of contrasts and thelike to be able to broadly see the different constituents of the tissue.An example would be to use a silver stain to highlight axons orreticulin fibers, or use stains to stain fibrous tissues, glycogen orcell cytoplasms. Special stains are cumbersome and often quite toxic.Some not necessarily limiting examples of special stains include:periodic acid - Schiff (PAS/D), Masson’s trichrome, Jones Silver H&E,Mucicarmine, Ziehl-Neelsen, and Elastic.

The terms permanent stain and special stain may sometimes be usedinterchangeably.

As used herein, the term first stain and second stain may referrespectively, for example, to a removable stain and a permanent stain,to a permanent stain and to a permanent stain, to a first permanentstain and to a second permanent stain, and to a first removable stainand to a second removable stain.

An aspect of some embodiments of the present invention relates tosystems, methods, devices, and code instructions (i.e., stored on a datastorage device and executable by one or more processors) for training avirtual stainer machine learning model. An imaging multi-record trainingdataset is created. A record of the imaging multi-record trainingdataset includes a first image of a sample of tissue of a subjectstained with a removable stain, and a ground truth indicated by a secondimage of the sample of tissue of the subject stained with a permanentstain. The second image may be created using the same sample of tissuedepicted in the first image after the first image is captured, bytreating the sample of tissue depicted in the first image to remove theremovable stain to create a cleared tissue sample. The permanent stainis applied to the cleared tissue sample to create a permanently stainedsample. The second image depicts the permanently stained sample. Thevirtual stainer machine learning model is trained on the imagingmulti-record training dataset. The trained virtual stainer machinelearning model generates a virtual image depicting the permanent stainin response to an input image depicting the removable stain.

An aspect of some embodiments of the present invention relates tosystems, methods, devices, and code instructions (i.e., stored on a datastorage device and executable by one or more processors) for generatingan image of virtually stained tissue. A target image of a sample oftissue of a subject stained with a removable stain is fed into a virtualstainer machine learning model. A synthetic image depicting the sampleof tissue stained with a permanent stain is obtained as an outcome ofthe virtual stainer machine learning model. The virtual stainer machinelearning model is trained as described herein.

An aspect of some embodiments of the present invention relates tosystems, methods, devices, and code instructions (i.e., stored on a datastorage device and executable by one or more processors) for training avirtual stainer machine learning model. An imaging multi-record trainingdataset is created. A record of the imaging multi-record trainingdataset includes a first image of a sample of tissue of a subjectstained with a permanent stain, and a ground truth indicated by a secondimage of the sample of tissue of the subject stained with a removablestain. The sample of tissue may be first stained with the removablestain to create a removable stain stained sample. The second image ofthe removable stain stained sample is captured prior to capture of thefirst image. The removable stain stained sample is treated to clear theremovable stain to create a cleared tissue sample. The permanent stainis applied to the cleared tissue sample to create a permanent stainstained sample. The first image depicting the permanent stain stainedsample is captured. The virtual stainer machine learning model istrained on the imaging multi-record training dataset. The trainedvirtual stainer machine learning model generates a virtual imagedepicting the removable stain in response to an input image depictingthe permanent stain.

An aspect of some embodiments of the present invention relates tosystems, methods, devices, and code instructions (i.e., stored on a datastorage device and executable by one or more processors) for generatingan image of virtually stained tissue. A target image of a sample oftissue of a subject stained with a permanent stain is fed into a virtualstainer machine learning model. A synthetic image depicting the sampleof tissue stained with a removable stain is obtained as an outcome ofthe virtual stainer machine learning model. The virtual stainer machinelearning model is trained as described herein.

At least some embodiments described herein address the technical problemof providing faster and/or better availability for permanent stains(e.g., special stains). Staining tissue samples with permanent stains,sometimes also referred to herein as special stains, is cumbersome andquite often toxic. As such, using such stains takes a significant amountof time, requires expert knowledge, requires special equipment, and/orrequires care in preventing toxicity. This makes it difficult to preparetissue samples stained with special stains. At least some embodimentsdescribed herein improve the technical field of machine learning, inparticular, the technical field of machine learning for generatingvirtual images. At least some embodiments described herein improve thetechnical field of medicine, in particular pathology, by providingimages of tissue stained with a permanent stain in response to an inputof tissue stained with a removable stain. At least some embodimentsdescribed herein address the previously mentioned technical problemand/or improve the previously mentioned technical field, by creating amulti-record training dataset, where a record of the multi-recordtraining dataset includes a first image and a second image serving asground truth, that are based on the same tissue sample. The tissuesample is first stained with a removable stain, optionally H&E. Thefirst image is captured. The removable stain is removed to obtain acleared tissue sample. The cleared tissue sample is stained with apermanent stain. The second image (i.e., ground truth) depicting thesame tissue sample stained with the permanent stain is captured. Thefirst and second images, which depict the same tissue, may be alignedusing an automated alignment process (e.g., based on optical flow, asdescribed herein), for example, to correct physical differences betweenthe two images resulting from the physical staining process. The virtualstainer machine learning model is trained on the multi-record trainingdataset.

At least some embodiments described herein address the technical problemof obtaining an image of tissue stained with a removable stain usingtissue that has already been stained with a permanent stain. Once thetissue has been stained with the permanent stain, the permanent staincannot be physically removed to enable staining the same tissue with theremovable stain. At least some embodiments described herein improve thetechnical field of machine learning, in particular, the technical fieldof machine learning for generating virtual images. At least someembodiments described herein improve the technical field of medicine, inparticular pathology, by providing images of tissue stained with aremovable stain in response to an input of tissue stained with apermanent stain. At least some embodiments described herein address thepreviously mentioned technical problem and/or improve the previouslymentioned technical field, by creating a multi-record training dataset,where a record of the multi-record training dataset includes a firstimage and a second image serving as ground truth, that are based on thesame tissue sample. The tissue sample is first stained with a removablestain, optionally H&E. The second image (i.e., ground truth) is capturedprior to the first image. The removable stain is removed to obtain acleared tissue sample. The cleared tissue sample is stained with apermanent stain. The first image depicting the same tissue samplestained with the permanent stain is captured after the second image hasbeen captured. The first and second images, which depict the sametissue, may be aligned using an automated alignment process (e.g., basedon optical flow, as described herein), for example, to correct physicaldifferences between the two images resulting from the physical stainingprocess. The virtual stainer machine learning model is trained on themulti-record training dataset.

At least some embodiments described herein address the technical problemof obtaining multiple different stained versions for the same tissuesample, in particular different permanent stains. Using standardapproaches, once a first permanent stain is applied to the tissue,another permanent stain cannot be applied to obtain a meaningful stain(i.e., excluding the case of sequential staining described herein).Since different permanent stains may depict different features (e.g.,biomarkers) of the tissue, using different permanent stains of the sametissue may help provide additional pathological data, for example, forimproving diagnosis and/or treatment of the subject. At least someembodiments described herein improve the technical field of medicine, inparticular pathology. At least some embodiments described herein addressthe previously mentioned technical problem and/or improve the previouslymentioned technical field, by providing multiple virtual stainer MLmodels (or a single main virtual stainer) trained on differentmulti-record training datasets, for providing multiple virtual images ofthe same tissue depicting different permanent stains, in response to aninput of an image of the same tissue (which may be stained using apermanent and/or removable stain).

An aspect of some embodiments of the present disclosure relates tosystems, methods, a computing device, and/or code instructions (storedon a memory and executable by one or more hardware processors) forautomatically creating a ground truth training dataset for training aground truth generating machine learning model (sometimes referred toherein as “Model G*”). The ground truth training dataset includesmultiple records, where each record includes a first image, a secondimage, and ground truth labels (e.g., manually entered by a user, and/orautomatically determined such as by a set of rules). The first andsecond images are of a same sample of tissue of a subject. The firstimage depicts a first group of biological objects (e.g., cells,nucleoli). The first image may be stained with a stain and/orilluminated with an illumination, designed to visually distinguish afirst biomarker. Alternatively, the first image may not be stainedand/or illuminated with a specific illumination, for example, abrightfield image. The second image may be stained with a sequentialstain over the first stain (when the first image depicts tissue stainedwith the first stain), stained with the first stain (when the firstimage depicts unstained tissue) and/or illuminated with an illumination,designed to visually distinguish a second biomarker(s) (when the firstimage depicts the first biomarker) and/or distinguish a biomarker(s)when the first image does not necessarily depict a specificbiomarker(s). The ground truth labels indicate respective biologicalobject categories. The ground truth labels may be manually provided by auser and/or automatically determined such as by a set of rules. Theground truth generating machine learning model is trained on the groundtruth training dataset.

As used herein, the term “unlabeled” with reference to the images refersto no annotations, which may be manually inputted by a user and/orautomatically generated (e.g., by the ground truth generator and/or rulebased approach, as described herein). The term “unlabeled” is not meantto exclude some staining approaches (e.g., fluorescence stain) which insome fields of biology may be referred to as “labels”. For example, inlive-cell image analysis. As such, images stained with stains such asfluorescence stains that are unannotated as referred to herein as“unlabeled”.

The ground truth generating machine learning model is used to generateground truth labels in response to an input of unlabeled first and/orsecond images. A synthetic training dataset may be created, where eachrecord of the synthetic training dataset includes the unlabeled firstand/or second images, labeled with the ground truth labels automaticallygenerated by the ground truth generating machine learning model. Recordsof the synthetic training dataset may be further labeled with a groundtruth indication of a diagnosis for the sample of tissue depicted in theunlabeled first and/or second images. The diagnosis may be obtained, forexample, using a set of rules (e.g., compute percentage of tumor cellsrelative to all cells), extracted from a pathology report, and/ormanually provided by a user (e.g., pathologist). The synthetic trainingdataset may be used to train a diagnosis machine learning model and/orbiological object machine learning model (sometimes referred to hereinas “Model G”).

In different implementations, to obtain a diagnosis, once the diagnosisand/or biological object machine learning model (Model G) is trained,the ground truth generating machine learning model (Model G*) is notnecessarily used for inference. A target unlabeled first and/or secondimage may be fed into the diagnosis and/or biological object machinelearning model to obtain the diagnosis. It is noted that alternativelyor additionally, a deterministic approach (i.e., non-ML model that isbased on a predefined process and not learned from the data) may be usedfor the biological object implementation.

In another implementation, during inference, a target unlabeled firstand/or second image is fed into the ground truth generating machinelearning model to obtain automatically generated labels. The labelledtarget first and/or second images may be analyzed to obtain a diagnosis,by feeding into the diagnosis machine learning model and/or applying theset of rules.

Virtual second images corresponding to the first image may besynthesized by a virtual stainer machine learning model (sometimesreferred to herein as “Model F”), and used as the second imagesdescribed herein. This enables using only first images without requiringphysical capturing second images which may require special sequentialstaining procedures and/or special illumination.

In another implementation, during inference, a combination of the firstimage and a second virtually stained image obtained from the virtualstainer in response to an input of the first image, may be fed into theground truth generating machine learning model to obtain annotations.The automatically annotated first and second images (i.e., first imageand virtual second image) may be fed in combination into the diagnosisand/or biological object machine learning model to obtain the diagnosis.Alternatively, in yet another implementation, the combination of thefirst image and a second virtually stained image are fed into thediagnosis and/or biological object machine learning model to obtain thediagnosis, i.e., the step of feeding into the ground truth generator maybe omitted.

Examples of biological objects (sometimes referred to herein as “targetmolecules”) include cells, for example, immune cells, red blood cells,malignant cells, non-malignant tumor cells, fibroblasts, epithelialcells, connective tissue cells, muscle cells, and neurite cells), andinternal/or part of cell structures, such as nucleus, DNA, mitochondria,and cell organelles. Biological objects may be a complex of multiplecells, for example, a gland, a duct, an organ or portion thereof such asliver tissue, lung tissue, and skin. Biological objects may includenon-human cells, for example, pathogens, bacteria, protozoa, and worms.Biological objects may sometimes be referred to herein as “target”, andmay include examples of targets described herein.

Examples of samples that are depicted in images include slides oftissues (e.g., created by slicing a frozen section, Formalin-FixedParaffin-Embedded (FFPE)) which may be pathological tissue, and livecell images. Other examples of the sample (sometimes referred to hereinas “biological sample”) are described herein.

The sample of tissue may be obtained intra-operatively, during forexample, a biopsy procedure, a FNA procedure, a core biopsy procedure,colonoscopy for removal of colon polyps, surgery for removal of anunknown mass, surgery for removal of a benign cancer, and/or surgery forremoval of a malignant cancer, surgery for treatment of the medicalcondition. Tissue may be obtained from fluid, for example, urine,synovial fluid, blood, and cerebral spinal fluid. Tissue may be in theform of a connected group of cells, for example, a histological slide.Tissue may be in the form of individual or clumps of cells suspendedwithin a fluid, for example, a cytological sample.

Other exemplary cellular biological samples include, but are not limitedto, blood (e.g., peripheral blood leukocytes, peripheral bloodmononuclear cells, whole blood, cord blood), a solid tissue biopsy,cerebrospinal fluid, urine, lymph fluids, and various externalsecretions of the respiratory, intestinal and genitourinary tracts,synovial fluid, amniotic fluid and chorionic villi.

Biopsies include, but are not limited to, surgical biopsies includingincisional or excisional biopsy, fine needle aspirates and the like,complete resections or body fluids. Methods of biopsy retrieval are wellknown in the art.

The biomarker(s) may refer to a physical and/or chemical and/orbiological feature of the biological object that is enhanced with thestain and/or illumination, for example, surface proteins. In someembodiments, the biomarker may be indicative of a particular cell type,for example a tumor cell or a mononuclear inflammatory cell.

Alternatively or additionally, the term “biomarker” may describe achemical or biological species which is indicative of a presence and/orseverity of a disease or disorder in a subject. Exemplary biomarkersinclude small molecules such as metabolites, and biomolecules such asantigens, hormones, receptors, and any other proteins, as well aspolynucleotides. Any other species indicative of a presence and/orseverity of medical conditions are contemplated.

The terms first image, image of a first type, image depicting one ormore first biomarkers (or biomarkers of a first type), are used hereininterchangeably. The first image depicts a sample of tissue of a subjectdepicting a first group of biological objects. The first image may beunstained and/or without application of special light, i.e., notnecessarily depicting a specific biomarker, for example, a brightfieldimage. Alternatively or additionally, the first image may be stainedwith a stain designed to depict a specific biomarker expressed by thefirst group, also sometimes referred to herein as a first biomarker.Exemplary stains include: immunohistochemistry (IHC), fluorescence,Fluorescence In Situ Hybridization (FISH) which may be made by Agilent®(e.g., seehttps://www(dot)agilent(dot)com/en/products/dako-omnis-solution-for-ihc-ish/define-your-fish,Hematoxylin and Eosin (H&E), PD-L1, Multiplex Ion Beam Imaging (MIBI),special stains manufactured by Agilent® (e.g., seehttps://www(dot)agilent(dot)com/en/product/special-stains), and thelike. Alternatively or additionally, the first image is captured byapplying a selected illumination, that does not necessarily physicallystain the sample (although physical staining may be performed), forexample, a color image (e.g., RGB), a black and white image,multispectral (e.g., Raman spectroscopy, Brillouin spectroscopy,second/third harmonic generation spectroscopy (SHG/THG)), confocal,fluorescent, near infrared, short wave infrared, and the like.Alternatively or additionally, the first image is captured by a labelmodality such as chromatin stained bright field image and/orfluorescence stained image. Alternatively or additionally, the firstimage is captured by a non-label process such as auto-fluorescenceand/or a spectral approach that emphasize naturally the biomarkerwithout the need of an external tagging stain. Alternatively oradditionally, the first image is captured by another non-label modality,for example, cross-polarization microscopy. It is noted that processesdescribed with reference to the first image are not necessarily limitedto the first image, and may be used for the second image or additionalimages, in different combinations. For example, the first image isprocessed using a non-label process and the second image is processedusing a label process.

The terms second image, image of a second type, image depicting one ormore biomarkers (used when the first image does not necessarily depictany specific biomarkers, such as for brightfield images), and imagedepicting one or more second biomarkers (or biomarkers of a secondtype), are used herein interchangeably. The second image depicts thesame tissue depicted in the first image, and visually distinguishes asecond group of biological objects. The second image may be stained witha stain designed to depict a specific biomarker expressed by the secondgroup. The second image may be stained with a sequential stain designedto depict the second biomarker(s), where the first image is stained witha first stain designed to depict the first biomarker(s), and the secondsequential stain is further applied to the sample stained with the firststain to generate a second image with both the first stain and thesecond sequential stain. Alternatively or additionally, the first imageis captured by applying a selected illumination, that does notnecessarily physically stain the sample (although physical staining maybe performed), for example, autofluorescence, and the second imagedepicts staining and/or another illumination.

The images may be obtained, for example, from an image sensor thatcaptures the images, from a scanner that captures images, from a serverthat stores the images (e.g., PACS server, EMR server, pathologyserver). For example, tissue images are automatically sent to analysisafter capture by the imager and/or once the images are stored afterbeing scanned by the imager.

The images may be whole slide images (WSI), and/or patches extractedfrom the WSI, and/or portions of the sample. The images may be of thesample obtained at high magnification, for example, for an objectivelens - between about 20X-40X, or other values. Such high magnificationimaging may create very large images, for example, on the order of GigaPixel sizes. Each large image may be divided into smaller sized patches,which are then analyzed. Alternatively, the large image is analyzed as awhole. Images may be scanned along different x-y planes at differentaxial (i.e., z axis) depth.

In some implementations, the first image may be stained with a firststain depicting a first type of biomarker(s), and the second image maybe further stained with a second stain depicting a second type ofbiomarker(s) which is different than the first type of biomarker(s),also referred to herein as a sequential stain. The second sequentialstain is a further stain of the tissue specimen after the first stainhas been applied. Alternatively or additionally, the second image may becaptured by a second imaging modality, which is different than a firstimaging modality used to capture the first image. The second imagingmodality may be designed to depict a second type of physicalcharacteristic(s) (e.g., biomarkers) of the tissue specimen that isdifferent than a first type of physical characteristic(s) of the tissuespecimens depicted by the first imaging modality. For example, differentimaging sensors that capture images at different wavelengths.

The first image and the second image may be selected as a set. Thesecond image and the first image are of the same sample of tissue. Thesecond image depicts biomarkers or second biomarkers when the firstimage does not explicitly depict the biomarkers (e.g., brightfield). Thesecond image may depict second biomarkers when the first image depictsfirst biomarkers. The second group of biological objects may bedifferent than the first group of biological objects, for example, thefirst group is immune cells, and the second group is tumor cells. Thesecond group of biological objects may be in the environment of thefirst group, for example, in near proximity, such as within a tumorenvironment. For example, immune cells located in proximity to the tumorcells. The second group of biological objects depicted in the secondimage may include a first sub-group of the first group of biologicalobjects of the first image, for example, the first group in the firstimage includes white blood cells, and the second group is macrophages,which are a sub-group of white blood cells. The second image may depicta second sub-group of the first group that is not expressing the secondbiomarkers, for example, in the case of sequential staining the secondsub-group is unstained by the second stain (but may be stained by thefirst stain of the first image). The second sub-group is excluded fromthe second group of biological structures. For example, in the exampleof the first image stained with a first stain that depicts white bloodcells, and the second image is stained with a sequential stain thatdepicts macrophages, other white blood cells that are non-macrophages(e.g., T cells, B cells) are not stained with the sequential stain.

It is noted that the first and second biomarkers may be expressed indifferent regions and/or compartments of the same object. For example,the first biomarker is expressed in the cell membrane and the secondbiomarker is expressed in the cell cytoplasm. In another example, thefirst biomarker and/or second biomarker may be expressed in a nucleus(e.g., compartment) of the cell.

Examples of first images are shown with respect to FIGS. 1A and 2A, andcorresponding second images depicting sequential staining are shown withrespect to FIGS. 1B and 2B.

In some implementations, the first image depicts the first group ofbiological objects presenting the first biomarker, and the second imagedepicts the second group of biological objects presenting the secondbiomarker. The second biomarker may be different than the firstbiomarker. The first image may depict tissue stained with a first staindesigned to stain the first group of biological objects depicting thefirst biomarker. The second image may depict a sequential stain, wherethe tissue stained with the first stain is further stained with a secondsequential stain designed to stain the second group of biologicalobjects depicting the second biomarker in addition to the first group ofbiological objects depicting the first biomarker (e.g., stained with thefirst stain).

In some implementations, the first image is captured with a firstimaging modality that applies a first specific illumination selected tovisually highlight the first group of biological objects depicting thefirst biomarker, for example, an auto-fluorescence imager, and aspectral imager. In another example, the first image is a brightfieldimage, i.e., not necessarily depicting a specific first biomarker. Thesecond image is captured with a second imaging modality that applies asecond specific illumination selected to visually highlight the secondgroup of biological objects depicting the second biomarker (or abiomarker in the case of the first image not necessarily depicting thefirst biomarker).

In some implementations, the first image is captured by a first imagingmodality, and the second image is captured by a second imaging modalitythat is different than the first imaging modality. Examples of the firstand second imaging modalities include a label modality and a non-labelmodality (or process). The label modality may be, for example, chromatinstained bright field image and/or fluorescence stained image. Thenon-label may be, for example, auto-fluorescence and/or a spectralapproach that emphasize naturally the biomarker without the need of anexternal tagging stain. Another example of the non-label modality iscross-polarization microscopy. In other examples, the first imagecomprises a brightfield image, and the second images comprises one orboth of: (i) spectral imaging image indicating the at least onebiomarker, and (ii) non-labelled image depicting the at least onebiomarker (e.g., auto-fluorescence, spectral approach, andcross-polarization microscopy). Alternatively or additionally, the firstimage is captured by applying a selected illumination, that does notnecessarily physically stain the sample (although physical staining maybe performed), for example, a color image (e.g., RGB), a black and whiteimage, multispectral (e.g., Raman spectroscopy, Brillouin spectroscopy,second/third harmonic generation spectroscopy (SHG/THG)), confocal,fluorescent, near infrared, short wave infrared, and the like.Alternatively or additionally, the first image is captured by a labelmodality such as chromatin stained bright field image and/orfluorescence stained image. Alternatively or additionally, the firstimage is captured by a non-label process such as auto-fluorescenceand/or a spectral approach that emphasize naturally the biomarkerwithout the need of an external tagging stain. Alternatively oradditionally, the first image is captured by another non-label modality,for example, cross-polarization microscopy.

As used herein, the term first and second images, or sequential imageand/or the term sequential stain is not necessarily limited to two. Itis to be understood that the approaches described herein with respect tothe first and second images may be expanded to apply to three or moreimages, which may be sequential images (e.g., sequential stains) and/orother images (e.g., different illuminations which are not necessarilysequentially applied in addition to previous illuminations). In anon-limiting example, the first image is a PD-L1 stain, the second imageis a sequential stain further applied to the sample stained with PD-L1,and the third image is a fluorescence image, which may be obtained ofone or more of: bright field image of the sample, of the first image, ofthe second image. Implementations described herein are described forclarity and simplicity with reference to the first and second images,but it is to be understood that the implementations are not necessarilylimited to the first and second images, and may be adapted to three ormore images.

An alignment process may be performed to align the first image and thesecond image with each other. The physical staining of the second imagemay physically distort the second image, causing a misalignment withrespect to the first image, even though the first image and second imageare of the same sample. The distortion may of the sample and thereforeof the image itself may occur, for example, from uncovering of a slideto perform an additional staining procedure (e.g., sequential staining).It is noted that only the image itself may be distorted due to multiplescanning operations, and the sample itself may remain undistorted. Thealignment may be performed with respect to identified biologicalobjects. In the aligned images, pixels of a certain biological object inone image map to the same biological object depicted in the other image.The alignment may be, for example, a rough alignment (e.g., where largefeatures are aligned) and/or a fine alignment (e.g., where fine featuresare aligned). The rough alignment may include an automated global affinealignment process that accounts for translation, rotation, and/oruniform deformation between the first and second images. The finealignment may be performed on rough aligned images. The fine alignmentmay be performed to account for local non-uniform deformations. The finealignment may be performed, for example, using optical flow and/ornon-rigid registration. The rough alignment may be sufficient forcreating annotated datasets, as described herein, however fine alignmentmay be used for the annotated datasets. The fine alignment may be usedfor creating a training dataset for training the virtual stainerdescribed herein. Examples of alignment approaches are described herein.

At least some implementations of the systems, methods, apparatus, and/orcode instructions described herein generate outcomes of ML models whichare used for biomarker and/or co-developed or companion diagnosis (CDx)discovery and/or development processes.

At least some implementations of the systems, methods, apparatus, and/orcode instructions described herein address the technical problem ofobtaining sufficient amount of ground truth labels for biologicalobjects depicted in images of samples of tissue for training a machinelearning model, for example, for identifying target biological objectsin target image(s) and/or generating a diagnosis for the target image(s)(e.g., treat subject with chemotherapy, medical condition diagnosis). Atleast some implementations of the systems, methods, apparatus, and/orcode instructions described herein improve the technical field ofmachine learning, by providing an automated approach for obtainingsufficient amount of ground truth labels for biological objects depictedin images of samples of tissue for training a machine learning model,for example, for identifying target biological objects in targetimage(s) and/or generating a diagnosis for the target image(s).

Training a machine learning model requires a large number of labelledimages. Traditionally, the labelling is performed manually. Thedifficulty is that the person qualified to perform the manual labor isgenerally a trained pathologist, which are in short supply and difficultto find to generate the large number of labelled images. Even when suchtrained pathologists are identified, the manual labelling is timeconsuming, since each image may contain thousands of biological objects(e.g., cells) of different types and/or at different states (e.g.,dividing, apoptosis, actively moving). Some types of biological objects(e.g., cells) are difficult to differentiate using the image, whichrequires even more time to evaluate. Moreover, the manual labelling isprone to error, for example, error in differentiating the differenttypes of biological objects.

In at least some implementations, a solution provided for the technicalproblem, and/or an improvement over the existing standard manualapproaches, is to provide a second (or more) image in addition to thefirst image, where the second image and the first image depict the samesample, and may be aligned with each other, such that specificbiological objects in one tissue are directly mapped to the samespecific biological objects depicted in the other image. In someimplementations, the solution is based on a sequential second image of asecond type (e.g., using a second sequential stain in addition to afirst stain) in addition to the original first image of a first type(e.g., the first stain without the second sequential stain), forpresentation on a display to the manual annotator. The manual annotationperformed using the first and second images increases accuracy ofdifferentiating between the different types of biological objects. Thenumber of biological objects that are manually annotated issignificantly smaller than would otherwise be required for generating aground truth data for a desired target performance level. A ground truthgenerator machine learning model is trained on a training dataset ofsets (e.g., pairs or greater number) of images (i.e., the first imageand the second image) labelled with the manual annotations as groundtruth and/or labelled with ground truth annotations that areautomatically obtained such as by a set of rules. In someimplementations, the training data includes one of the images andexclude the other (e.g., only first images excluding second images),where the included images are labelled with annotations generated usingboth the first and second images. The ground truth generator is fedunlabeled sets of images (which may be new images, or patches of theimages used in the training dataset which have not been manuallyannotated) and/or fed single images such as of the first type (e.g.,according to the structure of records of the training dataset) andautomatically generates an outcome of annotations for the biologicalobjects depicted in the input sets of images and/or input single imagesuch as the first image. The ground truth generator automaticallygenerates labelled sample images, in an amount that is sufficient fortraining a machine learning model, such as to perform at a targetperformance metric. The sample imaged labeled with the automaticallygenerated labels may be referred to herein as synthetic records. Thesynthetic records may further be labelled, for example, with anindication of diagnosis, and used to train a diagnosis machine learningmodel that generates an outcome of a diagnosis for an input of anautomatically labelled target image (e.g., labelled by the ground truthgenerator).

At least some implementations of the systems, methods, apparatus, and/orcode instructions described herein address the technical problem ofproviding additional sequential images depicting additional biomarkers,in addition to first images of a first type depicting first type ofbiomarker(s). At least some implementations of the systems, methods,apparatus, and/or code instructions described herein improve thetechnology of analysis of image of samples of tissue by providingsequential images. The sequential image is of the same physical tissueslice depicted in the first image. The additional images depictingadditional biomarkers may assist pathologists in making a diagnosis,and/or classifying biological objects (e.g., cells) which may bedifficult to make using the first images alone. Traditionally,pathologists do not have access to sequential images depictingadditional biomarkers, and are required to make a diagnosis and/orclassify biological objects on the basis of the first image alone. Oncea sample of tissue is stained the first time, in many cases anotherstain cannot be applied using traditional methods, as applying a secondstain over the first stain may destroy the tissue sample, and/or renderthe tissue sample illegible for reading by a pathologist. Using thefirst image alone is more difficult, more prone to error, and/orrequires higher levels of training. Using sequential images increasesthe amount of data available, including increased accuracy of assigningspecific classification categories to objects depicted in the images.The additional data and/or increased classification categories may beused to train a machine learning model.

As described herein, sequential images may be prepared and presented ona display. However, some facilities may be unable to generate their ownsequential images, such as due to lack of availability of equipmentand/or lack of know-how. In at least some implementations, the technicalsolution to the technical problem is to train a virtual image machinelearning model (sometimes referred to herein as a virtual stainer),which generates a virtual second sequential image in response to aninput of a first image. The virtual second sequential image is similarto a second image that would otherwise be prepared by physically addingthe second sequential stain, and/or by capturing the second image usinga second sequential imaging modality. The virtual second sequentialimage is generated by the virtual stainer model without requiringphysically performing the second sequential stain and/or withoutcapturing the second image with the second imaging modality.

At least some implementations of the systems, methods, apparatus, and/orcode instructions described herein improve the technology of machinelearning, by training and providing the virtual stainer machine learningmodel.

The virtually created second sequential images may be used as-is, bypresenting the virtually created second sequential image on a displayoptionally beside the first image, to assist the pathologist in makingthe diagnosis and/or labelling the biological objects depicted therein.Moreover, the virtually created second sequential images may be used inother processes described herein, for example, for creating the secondimages of the set that is fed into the ground truth generator. Inanother example, the diagnosis machine learning model training mightrequire sets of labelled patches, rather than only labelled firstimages. In those cases, when a target first image is obtained, thetarget second image may be automatically created by feeding the targetfirst image into the virtual stainer. The set of the target first imageand the target second image created by the virtual stainer may then befed into the ground truth generator to obtain labels, and then fed as aset into the diagnosis machine learning model to obtain a diagnosis.This enables using the diagnosis machine learning model trained on setsof ground truth images without requiring physically generating thesecond image by staining and/or by capturing using a second imagingmodality that applies a specific illumination.

Another technical challenge lies in the inability to obtain a secondsequential image of the first image, where the second image and thefirst image are of the same physical tissue slice specimen, and wherethe second image depicts a second type of biomarker(s) that aredifferent than the first type of biomarker(s). Using standardapproaches, images of neighboring slices may be used to help make thediagnosis and/or classify biological objects in a main image. The imagesof the neighboring slices are usually of the same first type of image,such as stained with the same stain as the first image. The neighboringslices are created by taking the tissue sample, physically slicing thesample into parallel physical slices, and creates slides of each slice.However, since the distance between the different slices is usuallylarger than size of cells, most cells will appear only in one slice, andnot in neighboring slices. The amount of information available fromneighboring slices is therefore limited. Once the slice is stained usinga standard staining approach, no additional information may be extractedusing standard approaches, i.e., there are no standard approaches forstaining with another stain over the first stained slide that depictsbiomarkers that are different than the biomarkers depicted in the firststained slide, and/or there are no standard approaches for imaging thefirst stained slide with another imaging modality that depicts differentbiomarkers. The virtually created second sequential image providesadditional visual data for the pathologist, depicting additionalbiomarkers in the same physical slice of tissue, in addition to thefirst biomarkers in the first stained same physical slice of tissue.Therefore allowing the pathologist to view information about two (ormore) bio-marker for each cell.

At least some implementations of the systems, methods, apparatus, and/orcode instructions described herein address the technical problem ofclassifying biological objects depicted in images of tissue sliceswithout relying on manual annotated data for training a machine learningmodel. Traditionally, a trained pathologist is required to manuallylabel biological objects. The trained pathologist may not be available,and/or may be limited in the amount of data that they can annotate. Atleast some implementations of the systems, methods, apparatus, and/orcode instructions described herein improve the technology of imageanalysis, by automatically classifying biological objects depicted inimages of tissue slices without relying on manual annotated data fortraining a machine learning model.

The second sequential image, which may be the virtually created secondsequential images, is used for classification of segmented biologicalobjects. In some implementations, the virtually created secondsequential image is created by the virtual stainer from an input firstimage. Target biological objects of the second sequential image,optionally the virtually created second sequential image, are segmentedby a segmentation process, for example nuclei are segmented. A set ofrules is applied to pixel values of colors specific to the secondsequential image, optionally the virtually created second sequentialimage, for mapping the pixel values using a set of rules or othermapping function to classification categories indicative of labels forthe segmented biological objects. The set of rules and/or other mappingfunction may be used instead of a machine learning model trained onmanually labelled images. The set of rules may be used, for example,when the segmented biological objects are stained with colors specificto the virtually created second sequential image, which may or may notalso include colors specific to the first type of image optionallystained with a first stain. For example, when the first stain is brown,and the second sequential stain is magenta (e.g., as described herein),the set of rules may map for example pixel intensity values indicatingmagenta to a specific classification category. The set of rules may beset, for example, based on observation by Inventors that the specificclassification category has pixel values indicating presence of thesecond sequential stain.

At least some implemenations described herein provide for robust and/orscalable solutions for tissue staining, for detectable reagents capableof serving as substrates of an enzyme with peroxidase activity, and/orfor detecting multiple target molecules in biological samples comprisingcells. There is also a need for robust and scalable solutionsimplementing annotation data collection and autonomous annotation.According to the present In at least some implemenations describedherein, teachings, methods, systems, and apparatuses are disclosed forimplementing sequential imaging of biological samples for generatingtraining data for developing deep learning based models for imageanalysis, for cell classification, for feature of interestidentification, and/or for virtual staining of biological samples Beforeexplaining at least one embodiment of the invention in detail, it is tobe understood that the invention is not necessarily limited in itsapplication to the details of construction and the arrangement of thecomponents and/or methods set forth in the following description and/orillustrated in the drawings and/or the Examples. The invention iscapable of other embodiments or of being practiced or carried out invarious ways.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, and any suitable combination of theforegoing. A computer readable storage medium, as used herein, is not tobe construed as being transitory signals per se, such as radio waves orother freely propagating electromagnetic waves, electromagnetic wavespropagating through a waveguide or other transmission media (e.g., lightpulses passing through a fiber-optic cable), or electrical signalstransmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user’scomputer, partly on the user’s computer, as a stand-alone softwarepackage, partly on the user’s computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user’s computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Reference is now made to FIG. 12 , which is a block diagram ofcomponents of a system 1200 for training ML models that analyze firstand second images of a sample of tissue and/or for inference of firstand second images of a sample of tissue using the ML models, inaccordance with some embodiments of the present disclosure. System 1200may be an alternative to, and/or combined with (e.g., using one or morecomponents) the system described with reference to FIG. 3 , FIG. 10 ,and/or FIG. 11 .

System 1200 may implement the acts of the method described withreference to FIGS. 23-30 and/or FIGS. 13-17 and/or FIGS. 4A-4C, 5A-5N,6A-6E, 7A-7D, 9A-9E, optionally by a hardware processor(s) 1202 of acomputing device 1204 executing code instructions 1206A and/or 1206Bstored in a memory 1206.

Computing device 1204 may be implemented as, for example, a clientterminal, a server, a virtual server, a laboratory workstation (e.g.,pathology workstation), a procedure (e.g., operating) room computerand/or server, a virtual machine, a computing cloud, a mobile device, adesktop computer, a thin client, a Smartphone, a Tablet computer, alaptop computer, a wearable computer, glasses computer, and a watchcomputer. Computing 1204 may include an advanced visualizationworkstation that sometimes is implemented as an add-on to a laboratoryworkstation and/or other devices for presenting images of samples oftissues to a user (e.g., pathologist).

Different architectures of system 1200 based on computing device 1204may be implemented, for example, central server based implementations,and/or localized based implementation.

In an example of a central server based implementation, computing device1204 may include locally stored software that performs one or more ofthe acts described with reference to FIGS. 23-30 and/or FIGS. 13-17and/or FIGS. 4A-4C, 5A-5N, 6A-6E, 7A-7D, 9A-9E, and/or may act as one ormore servers (e.g., network server, web server, a computing cloud,virtual server) that provides services (e.g., one or more of the actsdescribed with reference to FIGS. 23-30 and/or FIGS. 13-17 and/or FIGS.4A-4C, 5A-5N, 6A-6E, 7A-7D, 9A-9E) to one or more client terminals 1208(e.g., remotely located laboratory workstations, remote picturearchiving and communication system (PACS) server, remote electronicmedical record (EMR) server, remote image storage server, remotelylocated pathology computing device, client terminal of a user such as adesktop computer) over a network 1210, for example, providing softwareas a service (SaaS) to the client terminal(s) 1208, providing anapplication for local download to the client terminal(s) 1208, as anadd-on to a web browser and/or a tissue sample imaging viewerapplication, and/or providing functions using a remote access session tothe client terminals 1208, such as through a web browser. In oneimplementation, multiple client terminals 1208 each obtain images of thesamples from different imaging device(s) 1212. Each of the multipleclient terminals 1208 provides the images to computing device 1204. Theimages may be of a first image type (e.g., stained with the first staindepicting a first biomarker, imaged with a specific illumination, and/orbright field) and/or second image type (e.g., sequential image stainedwith the sequential stain depicting the second biomarker, stained with anon-sequential stain, and/or imaged with a different specificillumination). The images may be unlabeled, or may include patches withlabelled biological objects, for example, for training a ground truthannotator machine learning model, as described herein. Computing devicemay feed the sample image(s) into one or more machine learning model(s)1222A to obtain an outcome, for example, a virtual image, automaticlabels assigned to biological objects depicted in the image(s), and/orother data such as metrics and/or other clinical scores (e.g., computedpercentage of specific cells relative to total number of cells), whichmay be used for diagnosis and/or treatment (e.g., selecting subjects fortreatment with chemotherapy). The outcome obtained from computing device1204 may be provided to each respective client terminal 1208, forexample, for presentation on a display and/or storage in a local storageand/or feeding into another process such as a diagnosis application.Training of machine learning model(s) 1222A may be centrally performedby computing device 1204 based on images of samples and/or annotation ofdata obtained from one or more client terminal(s) 1208, optionallymultiple different client terminals 1208, and/or performed by anotherdevice (e.g., server(s) 1218) and provided to computing device 1204 foruse.

In a local based implementation, each respective computing device 1204is used by a specific user, for example, a specific pathologist, and/ora group of users in a facility, such as a hospital and/or pathology lab.Computing device 1204 receives sample images from imaging device 1212,for example, directly, and/or via an image repository 1214 (e.g., PACSserver, cloud storage, hard disk). Received images may be of the firstimage type and/or second image type. The raw images may be presented ona display 1226 associated with computing device 1204, for example, formanual annotation by a user, for training the ground truth generatormachine learning model, as described herein. Images may be locally fedinto one or more machine learning model(s) 1222A to obtain an outcome.The outcome may be, for example, presented on display 1226, locallystored in a data storage device 1222 of computing device 1204, and/orfed into another application which may be locally stored on data storagedevice 1222. Training of machine learning model(s) 1222A may be locallyperformed by each respective computing device 1204 based on images ofsamples and/or annotation of data obtained from respective imagingdevices 1212, for example, different users may each train their own setof machine learning models 1222A using the samples used by the user,and/or different pathological labs may each train their own set ofmachine learning models using their own images. For example, apathologist specializing in analyzing bone marrow biopsy trains MLmodels designed to annotated and/or infer bone marrow images. Anotherlab specializing in blood smears trains ML models designed to annotatedand/or infer blood smear images. In another example, trained machinelearning model(s) 1222A are obtained from another device, such as acentral server.

Computing device 1204 receives images of samples, captured by one ormore imaging device(s) 1212. Exemplary imaging device(s) 1212 include: ascanner scanning in standard color channels (e.g., red, green blue), amultispectral imager acquiring images in four or more channels, aconfocal microscope, a black and white imaging device, and an imagingsensor. Imaging device 1212 captures at least the first image describedherein. The second image described herein may be captured by imagingdevice 1212, another imaging device, and/or synthesized by feeding thefirst image into the virtual stainer, as described herein.

Optionally, one or more staining devices 1226 apply stains to the samplewhich is then imaged by imaging device 1212. Staining device 1226 mayapply the first stain, and optionally the second stain, which may be thesequential stain, as described herein.

Imaging device(s) 1212 may create two dimensional (2D) images of thesamples, optionally whole slide images.

Images captured by imaging machine 1212 may be stored in an imagerepository 1214, for example, a storage server (e.g., PACS, EHR server),a computing cloud, virtual memory, and a hard disk.

Training dataset(s) 1222B may be created based on the captured images,as described herein.

Machine learning model(s) 1222A may be trained on training dataset(s)1222B, as described herein.

Exemplary architectures of the machine learning models described hereininclude, for example, statistical classifiers and/or other statisticalmodels, neural networks of various architectures (e.g., convolutional,fully connected, deep, encoder-decoder, recurrent, graph), supportvector machines (SVM), logistic regression, other regressors, k-nearestneighbor, decision trees, boosting, random forest, a regressor, and/orany other commercial or open source package allowing regression,classification, dimensional reduction, supervised, unsupervised,semi-supervised or reinforcement learning. Machine learning models maybe trained using supervised approaches and/or unsupervised approaches.

Machine learning models described herein may be fine turned and/orupdated. Existing trained ML models trained for certain types of tissue,such as bone marrow biopsy, may be used as a basis for training other MLmodels using transfer learning approaches for other types of tissue,such as blood smear. The transfer learning approach of using an existingML model may increase the accuracy of the newly trained ML model and/orreduce the size of the training dataset for training the new ML model,and/or reduce the time and/or reduce the computational resources fortraining the new ML model, over standard approaches of training the newML model ‘from scratch’.

Computing device 1204 may receive the images for analysis from imagingdevice 1212 and/or image repository 1214 using one or more imaginginterfaces 1220, for example, a wire connection (e.g., physical port), awireless connection (e.g., antenna), a local bus, a port for connectionof a data storage device, a network interface card, other physicalinterface implementations, and/or virtual interfaces (e.g., softwareinterface, virtual private network (VPN) connection, applicationprogramming interface (API), software development kit (SDK)).Alternatively or additionally, Computing device 1204 may receive theslide images from client terminal(s) 1208 and/or server(s) 1218.

Hardware processor(s) 1202 may be implemented, for example, as a centralprocessing unit(s) (CPU), a graphics processing unit(s) (GPU), fieldprogrammable gate array(s) (FPGA), digital signal processor(s) (DSP),and application specific integrated circuit(s) (ASIC). Processor(s) 1202may include one or more processors (homogenous or heterogeneous), whichmay be arranged for parallel processing, as clusters and/or as one ormore multi core processing units.

Memory 1206 (also referred to herein as a program store, and/or datastorage device) stores code instruction for execution by hardwareprocessor(s) 1202, for example, a random access memory (RAM), read-onlymemory (ROM), and/or a storage device, for example, non-volatile memory,magnetic media, semiconductor memory devices, hard drive, removablestorage, and optical media (e.g., DVD, CD-ROM). Memory 1206 stores code1206A and/or training code 1206B that implements one or more acts and/orfeatures of the method described with reference to FIGS. 23-30 and/orFIGS. 13-17 and/or FIGS. 4A-4C, 5A-5N, 6A-6E, 7A-7D, 9A-9E.

Computing device 1204 may include a data storage device 1222 for storingdata, for example, machine learning model(s) 1222A as described herein(e.g., ground truth generator, virtual stainer, biological objectmachine learning model, and/or diagnosis machine learning model) and/ortraining dataset 1222B for training machine learning model(s) 1222A(e.g., ground truth training dataset, synthetic training dataset,biological object category training dataset, and/or imaging dataset), asdescribed herein. Data storage device 1222 may be implemented as, forexample, a memory, a local hard-drive, a removable storage device, anoptical disk, a storage device, and/or as a remote server and/orcomputing cloud (e.g., accessed over network 1210). It is noted thatexecution code portions of the data stored in data storage device 1222may be loaded into memory 1206 for execution by processor(s) 1202.Computing device 1204 may include data interface 1224, optionally anetwork interface, for connecting to network 1210, for example, one ormore of, a network interface card, a wireless interface to connect to awireless network, a physical interface for connecting to a cable fornetwork connectivity, a virtual interface implemented in software,network communication software providing higher layers of networkconnectivity, and/or other implementations. Computing device 1204 mayaccess one or more remote servers 1218 using network 1210, for example,to download updated versions of machine learning model(s) 1222A, code1206A, training code 1206B, and/or the training dataset(s) 1222B.

Computing device 1204 may communicate using network 1210 (or anothercommunication channel, such as through a direct link (e.g., cable,wireless) and/or indirect link (e.g., via an intermediary computingdevice such as a server, and/or via a storage device) with one or moreof:

-   Client terminal(s) 1208, for example, when computing device 1204    acts as a server providing image analysis services (e.g., SaaS) to    remote laboratory terminals, as described herein.-   Server 1218, for example, implemented in association with a PACS    and/or electronic medical record, which may store images of samples    from different individuals (e.g., patients) for processing, as    described herein.-   Image repository 1214 that stores images of samples captured by    imaging device 1212.

It is noted that imaging interface 1220 and data interface 1224 mayexist as two independent interfaces (e.g., two network ports), as twovirtual interfaces on a common physical interface (e.g., virtualnetworks on a common network port), and/or integrated into a singleinterface (e.g., network interface).

Computing device 1204 includes or is in communication with a userinterface 1226 that includes a mechanism designed for a user to enterdata (e.g., provide manual annotation of biological objects) and/or viewdata (e.g., virtual images, captured images). Exemplary user interfaces1226 include, for example, one or more of, a touchscreen, a display, akeyboard, a mouse, and voice activated software using speakers andmicrophone.

Reference is now made to FIG. 13 , which is a flowchart of a method ofautomatically generating a virtual stainer machine learning model (alsoreferred to herein as virtual stainer) that generates an outcome of asecond image in response to an input of a first image, in accordancewith some embodiments of the present disclosure. The virtual stainercreates a virtual second image from a target first image, making itunnecessary to physically create the second image, for example, thesequential stain does not need to be applied to the sample stained withthe first sample, and/or the illumination does not need to be applied.The virtual second image may be presented on a display, for example,alongside the first image. A user (e.g., pathologist) may use the firstimage and/or the virtual second image during the manual process ofviewing and/or reading the sample, such as for creating the pathologicalreport. The presentation of the virtual second image may be a standaloneprocess, and/or in combination with other processes described herein.Other exemplary approaches for training the virtual stainer aredescribed with reference to FIGS. 7A, 7C, and/or 7D. Examples of inputfirst images and virtual second images created by the virtual stainerare described with reference to FIGS. 8A-8O.

Referring now back to FIG. 13 , at 1302, a first image of a sample oftissue of a subject depicting a first group of biological objects, isaccessed.

At 1304, a second image of the sample of tissue depicting a second groupof biological objects, is accessed.

At 1306, the first image and second image, which depict the same tissue,may be aligned. Alignment may be performed when the second image is notdirectly correlated with the first image. For example, physicallyapplying the sequential stain to the sample of tissue already stainedwith the first stain may physical distort the tissue, creating themisalignment between the first and second images. The second image mayinclude local non-uniform deformations. For example, the alignment mayalign pixels depicting cells and/or pixels depicting nucleoli of thesecond image with the corresponding pixels depicting the same cellsand/or the same nucleoli of the first image.

An exemplary alignment process is now described. Biological featuresdepicted in the first image are identified, for example, by asegmentation process that segments target feature (e.g., cells,nucleoli), such as a trained neural network (or other ML modelimplementation) and/or using other approaches such as edge detection,histograms, and the like. The biological features may be selected as thebiological objects which are used for example, for annotating the imageand/or for determining a diagnosis for the sample. Biological featuresdepicted in the second image that correspond to the identifiedbiological features of the first image are identified, for example,using the approach used for the first image. An alignment process, forexample, an optical flow process and/or non-rigid registration process,is applied to the second image to compute an aligned image. Thealignment process (e.g., optical flow process and/or non-rigidregistration process) aligns pixels of the second image to correspondingpixels of the first image, for example, using optical flow and/ornon-rigid registration computed between the biological features of thesecond image and the biological features of the first image. In someimplementations, the alignment is a two stage process. A first coursealignment brings the two images to a cell-based alignment, and thesecond stage, which uses non-rigid registration, provides pixel-perfectalignment.

The first stage may be implemented using affine transform and/or scaletransform (to compensate for different magnifications) followedEuclidian/rigid transform At 1308, an imaging training dataset iscreated. The imaging training dataset may be multi-record, where aspecific record includes the first image, and a ground truth isindicated by the corresponding second image, optionally the alignedimage.

At 1310, a virtual stainer machine learning model is trained on theimaging training dataset. The virtual stainer ML model may beimplemented, for example, using a generative adversarial network (GAN)implementation where a generative network is trained to generatecandidate virtual images that cannot be distinguished from real imagesby a discriminative network, and/or based on the approaches describedherein, for example, with reference to FIGS. 7C and/or 7D.

Reference is now made to FIG. 14 , which is a flowchart of a method ofautomatically generating an annotated dataset of an image of a sample oftissue, in accordance with some embodiments of the present disclosure.The annotated dataset may be used to train a machine learning model, asdescribed herein.

Referring now back to FIG. 14 , at 1402, a first image of a sample oftissue of a subject depicting a first group of biological objects, isaccessed. The first image is unlabeled.

At 1404, the first image may be fed into the virtual stainer (e.g., asdescribed herein, for example, trained as described with reference toFIG. 13 ) to obtain the second image. The virtual stainer may be used,for example, when second images are not available, for example, infacilities where sequential staining cannot be performed, and/or byindividual pathologists that do not have the equipment to generatesecond images.

Alternatively, second images are available, in which case the virtualstainer is not necessarily used.

At 1406, the second image of the sample of tissue depicting a secondgroup of biological objects, is accessed. The second image may be thevirtual image created by the virtual stainer, and/or a captured image ofthe sample (e.g., sequentially stained sample, specific illuminationpattern applied). The second image is unlabeled.

At 1408, ground truth labels are manually provided by a user. Forexample, the user uses a graphical user interface (GUI) to manuallylabel biological objects, such as with a dot, line, outline,segmentation, and the like. Each indication represents a respectivebiological object category, which may be selected from one or morebiological object categories. For example, the user may mark macrophageswith blue dots, and mark tumor cells with red dots. Users may markbiological object members of the first group of the first and/or thesecond group of the second image.

Alternatively or additionally, as discussed herein, ground truth labelsmay be obtained directly from the second image by applying an imagemanipulation process. For example, segmenting the second stain on thesecond image and mapping the segmentation results to the biologicalobject. In such case, 1408 and 1410 might be considered as a single stepgenerating ground truth using a rule-based approach applied on the setof images.

Presenting the user with both the first image and second image may aidthe user to distinguish between biological objects. For example,biological objects stained with the first stain but not with thesequential stain are of one category, and biological objects stainedwith the sequential stain (in addition to the first stain) are ofanother category.

The ground truth labels may be applied to biological objects of thefirst image and/or second image. The labelling may be mapped to thebiological objects themselves, rather to the specific image, since thesame biological objects are depicted in both the first and secondimages. Alternatively or additionally, the labelling is mapped to one orboth images, where a label applied to one image is mapped to the samebiological object in the other image. Ground truth labels of biologicalobjects in the second image (e.g., presenting the second biomarker) maybe mapped to corresponding non-labeled biological objects of the firstimage (e.g., presenting the first biomarker).

It is noted that the user may apply ground truth labels to a portion ofthe biological objects depicted in the image(s). For example, when theimage(s) is a whole slide image, a portion of the biological objects maybe marked, and another portion remains unlabeled. The image may bedivided into patches. Annotations of ground truth labels may be appliedto objects in some patches, while other patches remain unlabeled.

At 1410, a ground truth training dataset is created. The ground truthtraining dataset may be multi-record. Different implementations of thetraining dataset may be created for example:

Each record includes a set of images, optionally a pair (or more) of thefirst type and second type, annotated with the ground truth labels. Eachlabel may be for individual biological objects, which appear in both thefirst and second images.

Each record includes only the first images, annotated with the groundtruth labels. Second images are excluded.

Each record includes only the second images, annotated with the groundtruth labels. First images are excluded.

At 1412, a ground truth generator machine learning model (also referredto herein as ground truth generator) is trained on the ground truthtraining dataset. The ground truth generator automatically generatesground truth labels selected from the biological object categories forbiological objects depicted in an input image corresponding to thestructure of the records used for training, for example, pair (or otherset) of images of the first type and the second type, only first images,and only second images.

Reference is now made to FIG. 15 , which is a flowchart of a method oftraining a biological object machine learning model and/or a diagnosismachine learning model using unlabeled images of samples of tissue, inaccordance with some embodiments of the present disclosure. The objectmachine learning model and/or a diagnosis machine learning model aretrained on ground truth annotations automatically generated for theinput unlabeled images by the ground truth generator.

Referring now back to FIG. 15 , at 1502, a first image of a sample oftissue of a subject depicting a first group of biological objects, isaccessed. The first image is unlabeled.

At 1504, the first image may be fed into the virtual stainer (e.g., asdescribed herein, for example, trained as described with reference toFIG. 13 ) to obtain the second image. The virtual stainer may be used,for example, when second images are not available, for example, infacilities where sequential staining cannot be performed, and/or byindividual pathologists that do not have the equipment to generatesecond images.

Alternatively, second images are available, in which case the virtualstainer is not necessarily used.

At 1506, the second image of the sample of tissue depicting a secondgroup of biological objects, may be accessed. The second image may bethe virtual image created by the virtual stainer, and/or a capturedimage of the sample (e.g., sequentially stained sample, specificillumination pattern applied). The second image is unlabeled.

At 1508, the unlabeled images(s) are fed into the ground truth generatormachine learning model. The ground truth generator may be trained asdescribed herein, for example, as described with reference to FIG. 14 .

The unlabeled images(s) fed into the ground truth generator correspondto the structure of the records of the ground truth training datasetused to train the ground truth generator. For example, the unlabeledimages(s) fed into the ground truth generator may include: a set ofunlabeled first and second images (where the second image may be anactual image and/or a virtual image created by the virtual stainer froman input of the first image), only first images (i.e., excluding secondimages), and only second images (i.e., excluding first images, where thesecond image may be an actual image and/or a virtual image created bythe virtual stainer from an input of the first image.

At 1510, automatically generated ground truth labels for the unlabeledfirst and/or second images are obtained from the ground truth generator.The labels may be for specific biological objects, without necessarilybeing associated with the first and/or second images in particular,since the same biological objects appear in both images and directlycorrespond to each other. It is noted that labels may be associated withfirst and/or second images in particular, where

The generated ground truth labels correspond to the ground truth labelsof the training dataset used to train the ground truth generator. Thelabels may be of object categories selected from multiple candidatecategories (e.g., different cell types are labelled) and/or of aspecific category (e.g., only cells types of the specific category arelabelled). Labels may be for biological objects depicting the firstbiomarker, and/or biological objects depicting the second biomarkerwhere the second biomarker is different than the first biomarker. Labelsmay be for any biological objects not necessarily depicting anybiomarker (e.g., brightfield images).

The generated ground truth labels may be implemented as, for example,segmentation of the respective biological objects, color coding of therespective biological objects, markers on the input images (e.g.,different colored dots marking the biological objects of differentcategories), a separate overlay image that maps to the input image(e.g., overlay of colored dots and/or colored segmentation region thatwhen placed on the input images corresponds to the locations of thebiological objects), metadata tags mapped to specific locations in theinput image, and/or a table of locations in the input image (e.g., x-ypixel coordinates) and corresponding label, and the like.

The ground truth labels may be of a regression type, for example, on ascale, which may be continuous and/or discrete. For example, a numericalscore on a scale of 1-5, or 1-10, or 0-1, or 1-100, or other values. Aregressor and/or other ML model architectures may be trained on suchground truth labels.

At 1512, a synthetic training dataset may be created from theautomatically generated ground truth labels. The synthetic trainingdataset may be multi-record. Different implementations of the synthetictraining dataset may be created for example:

Each synthetic record includes a set (e.g. pair) of the input images ofthe first type and second type, annotated with the automaticallygenerated ground truth labels. Each label may be for individualbiological objects, which appear in both the first and second images.Such implementation may be used, for example, for the ground truthgenerator bootstrapping itself, generating more ground truth from whicha ML model similar to the ground truth generator itself is trained.

Each synthetic record includes only the first input images, annotatedwith the automatically generated ground truth labels generated for thefirst and/or second images. Labels generated for the second images maybe mapped to corresponding biological objects on the first images.Second images are excluded.

Each synthetic record includes only the second input images, annotatedwith the automatically generated ground truth labels for the firstand/or second images. Labels generated for the first images may bemapped to corresponding biological objects on the second images. Firstimages are excluded.

At 1514, a biological object machine learning model may be trained onthe synthetic training dataset. The biological object machine learningmodel may generate an outcome of one or more biological objectcategories assigned to respective target biological objects depicted ina target image(s), for example, labels such as metadata tags and/orcolor coded dots assigned to individual biological objects. The targetinput image may correspond to the structure of the synthetic record usedto train the biological object machine learning model, for example, onlythe first image, only the second image, and a set (e.g., pair) of firstimage or second image where the second image may be a virtual imagecreated by the virtual stainer in response to an input of the first mageand/or the second image may be an actual captured image.

At 1516, alternatively or additionally to 1514, a diagnosis machinelearning model may be trained. The diagnosis machine learning modelgenerates an outcome indicating a diagnosis in response to an input ofimage(s).

Alternatively or additionally to a machine learning model, adeterministic (i.e., non-ML model that is based on a predefined processand not learned from the data) diagnosis based approach may be used.

The diagnosis machine learning model and/or the deterministic diagnosisapproach may, for example, count tumor cells classified as positive andtumor cells classified as negative and providing, for example, the ratiobetween the number of cells classified as positive and the number ofcells classified as negative, and/or the ratio of positive classifiedcells to the whole number of cells. Such diagnosis may be done, forexample, on top of the object machine learning model, for example, inthe context of PD-L1 scoring.

The diagnosis machine learning model may be trained on a biologicalobject category training dataset, which may include multiple records.Different implementations of the biological object category trainingdataset may be created for example:

The synthetic records of the synthetic training dataset are used torepresent the target input, labeled with a ground truth label of adiagnosis. The synthetic records may include: sets (e.g., pairs) offirst and second images, only first images, and only second images, asdescribed with reference to the structure of the synthetic records. Thediagnosis may be obtained, for example, manually from the user,extracted from a pathological report generated by a pathologist, and/orcomputed using a set of rules from the respective annotations, forexample, counting the number of total cells and computing a percentageof cells of a specific cell type.

Records may include images labelled with biological object categoriesobtained as an outcome of the biological object machine learning modelin response to input of the images (e.g., sets (e.g., pairs), only firstimages, only second images), labeled with a ground truth label of adiagnosis.

Reference is now made to FIG. 16 , which is a flowchart of analternative deterministic based approach for automatically annotatingbiological objects in images of samples of tissue, in accordance withsome embodiments of the present disclosure. The process described withreference to FIG. 16 may be an alternative to, and/or combined with, thenuclei detection system described herein. The approach described withreference to FIG. 16 may be an alternative to, and/or combined with oneor more features of the process described with reference to FIGS. 5A-5N.

Referring now back to FIG. 16 , at 1602, first images are accessed.

At 1604, second images are accessed. The second images may be actualimages, or virtual images obtained as an outcome of the virtual stainer(e.g., as described herein, such as trained as described with referenceto FIG. 13 ) fed an input of the first images.

The second image may be aligned to the first image, for example, usingoptical flow, non-rigid registration, and/or other approaches, asdescribed herein, for example, with reference to feature 1306 of FIG. 13.

At 1606, biological visual features of the first image are segmented.The segmented regions may correspond to biological objects that are forannotating, and/or used for determining a diagnosis, for example, cellsand/or nucleoli. The segmentation is performed using an automatedsegmentation process, for example, a trained segmentation neural network(or other ML model implementation), using edge detection approaches,using histogram computations, and the like.

At 1608, the segmentation computed for the first image are mapped to thesecond image. When the images are aligned, the segmentations of thefirst image may directly correspond to create segmentation on the secondimage.

At 1610, pixel values within and/or in proximity to a surrounding of therespective segmentation of the second image are computed. The pixelvalues indicate visual depiction of the second biomarker.

For example, when the first image is stained with a standard stain thatappears brown, and the second image is stained with a sequential stainthat appears to have a magenta color, pixel values within and/or inproximity to a surrounding of the respective segmentation of the secondimage indicating the magenta color are identified.

At 1612, the respective segmentation of the second image is classifiedby mapping the computed intensity value using a set of rules to aclassification category indicating a respective biological object type.For example, segmentations for which the magenta color is identified arelabelled with a certain cell type indicating cells that express thesecond biomarker.

At 1614, an annotation of the first and/or second images using theclassification categories is provided. Classification categoriesdetermined for segmentations of the second image may be used to labelcorresponding segmentations of the first image.

Reference is now made to FIG. 17 , which is a flowchart of a method ofobtaining a diagnosis for a sample of tissue of a subject, in accordancewith some embodiments of the present disclosure. The diagnosis may beobtained using a first image, without necessarily obtaining a secondimage by applying a sequential stain and/or specific illumination.

Referring now back to FIG. 17 , at 1702, a first target image of asample of tissue of a subject is accessed. The first target image isunlabeled. Additional exemplary details are described, for example, withreference to 1502 of FIG. 15 .

At 1704, the first target image may be fed into the virtual stainer(e.g., as described herein, for example, trained as described withreference to FIG. 13 ) to obtain the second target image. Alternatively,second target images are available, in which case the virtual stainer isnot necessarily used.

Additional exemplary details are described, for example, with referenceto 1504 of FIG. 15 .

At 1706, the second target image of the sample of tissue may beaccessed. The second target image is unlabeled. Additional exemplarydetails are described, for example, with reference to 1506 of FIG. 15 .

At 1708, the target images(s), i.e., the first target image and/orsecond target images, are fed into the ground truth generator machinelearning model. The ground truth generator may be trained as describedherein, for example, as described with reference to FIG. 14 .

Additional exemplary details are described, for example, with referenceto 1508 of FIG. 15 .

At 1710, automatically generated ground truth labels for the unlabeledfirst target image and/or second target image are obtained from theground truth generator. Additional exemplary details are described, forexample, with reference to 1510 of FIG. 15 .

At 1712, the target image(s) (i.e., the first and/or second images) maybe labelled with the automatically generated ground truth labelsobtained from the ground truth generator. The labelled target image(s)may be fed into a biological object machine learning model (e.g.,created as described herein, for example, with reference to 1514 of FIG.15 ). An outcome of respective biological object categories forrespective target biological objects depicted in the target image(s) isobtained from the biological object machine learning model.Alternatively, the target image(s) when non-labelled are fed into thebiological object machine learning model, to obtain an objectclassification outcome. The target image may be labelled with the objetclassification.

It is noted that the step of feeding into the biological object machinelearning model may be omitted in some implementations.

At 1714, the target image(s) may be labelled with the automaticallygenerated ground truth labels obtained from the ground truth generator,and/or labelled with the biological object classifications obtained fromthe biological object machine learning model.

The labelled target image(s) may be fed into a diagnosis machinelearning model (e.g., created as described herein, for example, withreference to 1516 of FIG. 15 . It is noted that the step of feeding intothe diagnosis machine learning model may be omitted in someimplementations.

Alternatively or additionally, one or more metrics (sometimes referredto herein as “clinical score”) are computed for the sample, using a setof rules, and/or using the diagnosis machine learning model. Metrics maybe computed, for example, per specific biological object category,and/or for multiple biological object categories. Metrics may becomputed, for example, for the sample as a whole, for the whole image(e.g., WSI), and/or for a patch of the image, and/or for a region(s) ofinterest. Examples of metrics include: Tumor Proportion Score (e.g., asdescribed herein), Combined Positive Score (e.g., as described herein),number of specific cells (e.g., tumor cells), percentage of certaincells (e.g., percent of macrophages out of depicted immune cells,percentage of viable tumor cells that express a specific biomarkerrelative to all viable tumor cells present in the sample, percentage ofcells expressing a specific biomarker, ratio between tumor and non-tumorcells), and other scores described herein.

It is noted that alternatively, rather than using the diagnosis machinelearning model, a set of rules may be used, and/or other deterministiccode, such as code that counts the number of biological objects inspecific categories and computes the percentage and/or other metrics.

At 1716, a diagnosis for the sample may be obtained as an outcome of thediagnosis machine learning model.

At 1718, the subject (whose sample is depicted in the target image(s))may be diagnosed and/or treated according to the obtained diagnosisand/or metric. For example, the subject may be administeredchemotherapy, may undergo surgery, may be instructed to watch-and-wait,and/or may be instructed to undergo additional testing.

Alternatively or additionally, new drugs and/or companion diagnosticsfor drugs may be developed using the diagnosis and/or computed metric.

Reference is now made to FIGS. 18A-C, which are schematics of exemplaryarchitectures of the virtual stainer machine learning model (sometimesreferred to herein as “Model F”), in accordance with some embodiments ofthe present disclosure. For example, an image of a slide stained withHematoxylin is fed into the virtual stainer, and an outcome of asequential stain (e.g., magenta appearance, as described herein) that isfurther applied over the Hematoxylin stain is generated.

Referring now to FIG. 18A, a first exemplary architecture 1802 of thevirtual stainer machine learning model is depicted. An input image 1804of spatial dimensions HxW is inputted into architecture 1802. Image 1804is fed into an encoder/decoder (ENC/DEC) network 1806, for example,based on a Unet architecture with a single channel output of the samespatial size as the input image, Intensity Map 1808. A ConvolutionalNeural Network (CNN) 1810 operates on the output of the encoder with acolor vector output, RGB vector 1812. Intensity map 1808 and colorvector 1812 are combined 1814 to produce a 3-channel image of thevirtual stain (virtual magenta) 1816. Predicted virtual stain 1816 isadded 1817 to input image 1804 to produce the output, a predictedvirtually stained image 1818. Architecture 1802 is trained with a loss1820, for example, MSE loss, between the virtually stained image 1818and its paired sequentially stained ground truth image 1822.

Referring now to FIG. 18B, a second exemplary architecture 1830 of thevirtual stainer machine learning model is depicted. Architecture 1830 issimilar to architecture 1802 described with reference to FIG. 18A, withthe adaptation of an added transformation of input images 1804 andground truth images 1822 to Optical Density (OD), by logarithmictransformation, using a bright-field (BF) to OD process 1832. Thetransformation to OD improves training of architecture 1830. When usingtrained model 1830 for inference, the inverse transformation, from ODspace (“log space”) to the regular image space (“linear space”), denotedOD to BF 1834 is applied to the output of model 1830 to generate output1836.

Referring now to FIG. 18C, a third exemplary architecture 1840 of thevirtual stainer machine learning model is depicted. Architecture 1834 issimilar to architecture 1830 described with reference to FIG. 18B, withthe adaptation of RGB color prediction network 1838 of FIG. 18B beingreplaced by a parameter vector of length three 1842, The (globally)optimal values for the virtual Magenta hue (R,G,B) 1842 are learntduring training and are kept fixed during inference.

In at least some implementations, the present disclosure providesdetectable reagents capable of serving as substrates of an enzyme withperoxidase activity, and describes their utility for detecting moleculartargets in samples. The present disclosure also provides methods fordetecting multiple target molecules in biological samples comprisingcells. Various embodiments also provide for implementing annotation datacollection and autonomous annotation, and, more particularly, tomethods, systems, and apparatuses for implementing sequential imaging ofbiological samples for generating training data for developing deeplearning based models for image analysis, for cell classification, forfeature of interest identification, and/or for virtual staining ofbiological samples.

A. Tissue Staining Methodology

A “moiety” is a portion of a molecule that retains chemical and/orphysical and/or functional features of the entire molecule, that arerelevant for performance of the chromogenic conjugates; e.g.,“peroxidase substrate moiety” is a portion of a molecule capable ofserving as substrate of an enzyme with peroxidase activity; “peroxidasemoiety” is a portion of a molecule that has inherent peroxidaseactivity, e.g., an enzyme.

A “target” is an object in a test sample to be detected by use of thepresent chromogenic conjugates and methods; Exemplary targets include,but are not limited to, chemical and biological molecules and structuressuch as cellular components. A target molecule includes, but is notlimited to: nuclear proteins, cytoplasmic proteins, membrane proteins,nuclear antigens, cytoplasmic antigens, polypeptides and membraneantigens, nucleic acid targets, DNA, RNA, nucleic acids inherent to theexamined biological material, and/or nucleic acids acquired in the formof viral, parasitic or bacterial nucleic acids. Embodiments of presenttargets are discussed herein.

The linker compound L comprises a chain of 5-29 interconnected atoms(correspondingly abbreviated “L5-L29”); wherein, in some preferredembodiments, the linker compound comprises two consecutive carbonsfollowed by an oxygen or nitrogen atom.

The term “detection method” can refer to immunohistochemistry (IHC), insitu hybridization (ISH), ELISA, Southern, Northern and Westernblotting.

The term, “Spectral characteristics” are characteristics ofelectromagnetic radiation emitted or absorbed due to a molecule ormoiety making a transition from one energy state to another energystate, for example from a higher energy state to a lower energy state.Only certain colors appear in a molecule’s or moiety’s emissionspectrum, since certain frequencies of light are emitted and certainfrequencies are absorbed. Spectral characteristics may be summarized orreferred to as the color of the molecule or moiety.

The term “CM” refers to the portion of the compound of Formula I asdescribed below other than the L-PS moieties.

The term “colorless” can refer to the characteristic where theabsorbance and emission characteristics of the compound changes, makingthe compound invisible to the naked eye, but not invisible to thecamera, or other methods of detection. It is known in the art thatvarious compounds can change their spectrum depending on conditions. Insome embodiments, changing the pH of the mounting medium can change theabsorbance and emission characteristics wherein a compound can go fromvisible in a basic aqueous mounting medium to invisible in an acidicmounting medium. This change is useful with regard to the detectablereagents as defined herein. Additionally, the chromogen as defined asFormula IV below (visible as yellow in aqueous mounting media), changesto Formula VII (invisible or colorless) when placed in organic mountingmedia.

Further, some embodiments use florescent compounds, which change theirabsorbance and emission characteristics through changing the pH of themounting medium, wherein, changing to a basic aqueous mounting mediumwherein the compound fluoresces, to an acidic mounting medium whereinthe compound does not. An example of such is Rhodamine spirolactam(non-fluorescent, acidic conditions) to spirolactone (highly fluorescentbasic conditions). Compounds include: fluorescein spirolactone,fluorescein spirolactams, Rhodamine Spirolactams and Rhodaminespirolactones. This includes Rhodamines and Fluoresceins that aremodified (derivatives). At acidic pH it is the less colored spiroformand at basic pH it is the “open form.”

Other colorless compounds that can be used in one embodiment of theinvention includes biotin as the CM, which is invisible to the nakedeye, but can be visualized at the appropriate time usingstreptavidin/HRP or an antibody/HRP (anti-biotin) and a peroxidasesubstrate that is either colored or fluorescent when precipitated.

The phrase, “visible in brightfield” means visible in parts of or thewhole brightfield spectra.

The phrase, “obtaining an image” includes obtaining an image from adigital scanner.

The term “detectable reagents” includes labels including chromogens.

B. Sequential Imaging of Biological Samples for Generating Training Datafor Developing Deep Learning Based Models for Image Analysis, for CellClassification, for Feature of Interest Identification, and/or forVirtual Staining of the Biological Sample

In various embodiments, a computing system may receive a first image ofa first biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample that has been processed witha first biological marker (which may include, without limitation, afirst stain, a first chromogen, or other suitable compound that isuseful for characterizing the first biological sample, or the like); andmay receive a second image of the first biological sample, the secondimage comprising a second FOV of the first biological sample that hasbeen processed with a second biological marker (which may include, butis not limited to, a second stain, a second chromogen, or other suitablecompound, or the like). The computing system may be configured to createa first set of image patches using the first image and the second image,wherein each image patch among the first set of image patches mayfurther correspond to a portion of the first image and a correspondingportion of the second image. The first set of image patches may comprisea first patch corresponding to the extracted portion of the first imageand a second patch corresponding to the extracted portion of the secondimage. In such cases, the first set of image patches may compriselabeling of instances of features of interest in the first biologicalsample that is based at least in part upon information contained in thefirst patch, information contained in the second patch, and/orinformation contained in one or more external labeling sources.

The computing system may utilize an AI system to train a first model(“Model G*”) to generate first instance classification of features ofinterest (“Ground Truth”) in the first biological sample, based at leastin part on the first set of image patches and the labeling of instancesof features of interest contained in the first set of image patches; andmay utilize the AI system to train a second AI model (“Model G”) toidentify instances of features of interest in the first biologicalsample, based at least in part on the first patch and the first instanceclassification of features of interest generated by Model G*.

According to some embodiments, the first biological sample may include,without limitation, any tissue or cellular sample derived from a livingorganism (including, but not limited to, a human tissue or cellularsample, an animal tissue or cellular sample, or a plant tissue orcellular sample, and/or the like) or an artificially produced tissuesample, and/or the like. In some instances, the features of interest mayinclude, but are not limited to, at least one of normal cells, abnormalcells, diseased cells, damaged cells, cancer cells, tumors, subcellularstructures, organs, organelles, cellular structures, pathogens,antigens, or biological markers, and/or the like.

In some embodiments, the first image may comprise highlighting oridentification of first features of interest in the first biologicalsample by the first biological marker that had been applied to the firstbiological sample. Similarly, the second image may comprise one ofhighlighting or identification of second features of interest in thefirst biological sample by the second biological marker that had beenapplied to the first biological sample in addition to highlighting oridentification of the first features of interest by the first biologicalmarker or highlighting of the second features of interest by the secondbiological marker without highlighting of the first features of interestby the first biological marker, the second features of interest beingdifferent from the first features of interest. In some cases, the secondbiological marker may be similar or identical to the first biologicalmarker but used to process the second features of interest or differentfrom the first biological marker, or the like.

Merely by way of example, in some cases, the first biological marker mayinclude, without limitation, at least one of Hematoxylin, Acridineorange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystalviolet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker (which may resultfrom the use of imaging techniques including, but not limited to, Ramanspectroscopy, near infrared (“NIR”) spectroscopy, autofluorescenceimaging, or phase imaging, and/or the like, and which may be used tohighlight features of interest without an external dye or the like),and/or the like. In some cases, the contrast when using label-freeimaging techniques may be generated without additional markers such asfluorescent dyes or chromogen dyes, or the like.

Likewise, the second biological marker may include, but is not limitedto, at least one of Hematoxylin, Acridine orange, Bismarck brown,Carmine, Coomassie blue, Cresyl violet, Crystal violet, DAPI, Eosin,Ethidium bromide intercalates, Acid fuchsine, Hoechst stain, Iodine,Malachite green, Methyl green, Methylene blue, Neutral red, Nile blue,Nile red, Osmium tetroxide, Propidium Iodide, Rhodamine, Safranine,PD-L1 stain, DAB chromogen, Magenta chromogen, cyanine chromogen, CD3stain, CD20 stain, CD68 stain, p40 stain, antibody-based stain, orlabel-free imaging marker, and/or the like.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image or a firstlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. Similarly, thesecond image may include, without limitation, one of a second set ofcolor or brightfield images, a second fluorescence image, a second phaseimage, or a second spectral image, and/or the like, where the second setof color or brightfield images may include, but is not limited to, asecond R image, a second G image, and a second B image, and/or the like.The second fluorescence image may include, but is not limited to, atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The second spectral imagemay include, but is not limited to, at least one of a second Ramanspectroscopy image, a second NIR spectroscopy image, a secondmultispectral image, a second hyperspectral image, or a second fullspectral image, and/or the like.

Alternatively, or additionally, the computing system may receive a firstimage of a first biological sample, the first image comprising a firstfield of view (“FOV”) of the first biological sample; and may identify,using a first AI model (“Model G”) that is generated or updated by atrained AI system, first instances of features of interest in the firstbiological sample, based at least in part on the first image and basedat least in part on training of Model G using a first patch and firstinstance classification of features of interest generated by a secondmodel (Model G*) that is generated or updated by the trained AI systemby using first aligned image patches and labeling of instances offeatures of interest contained in the first patch. In some cases, thefirst aligned image patches may comprise an extracted portion of firstaligned images, which may comprise a second image and a third image thathave been aligned. The second image may comprise a second FOV of asecond biological sample that has been processed with a first biologicalmarker (which may include, without limitation, a first stain, a firstchromogen, or other suitable compound that is useful for characterizingthe first biological sample, or the like), the second biological samplebeing different from the first biological sample. The third image maycomprise a third FOV of the second biological sample that has beenprocessed with a second biological marker (which may include, withoutlimitation, a second stain, a second chromogen, or other suitablecompound that is useful for characterizing the second biological sample,or the like). The second patch may comprise labeling of instances offeatures of interest as shown in the extracted portion of the secondimage of the first aligned images. The first image may comprisehighlighting of first features of interest in the second biologicalsample by the first biological marker that had been applied to thesecond biological sample. The second patch may comprise one ofhighlighting of second features of interest in the second biologicalsample by the second biological marker that had been applied to thesecond biological sample in addition to highlighting of the firstfeatures of interest by the first biological marker or highlighting ofthe second features of interest by the second biological marker withouthighlighting of the first features of interest by the first biologicalmarker.

Alternatively, or additionally, the computing system may receive firstinstance classification of features of interest (“Ground Truth”) in afirst biological sample that has been sequentially stained or processed,the Ground Truth having been generated by a trained first model (“ModelG*”) that has been trained or updated by an AI system, wherein theGround Truth is generated by using first aligned image patches andlabeling of instances of features of interest contained in the firstaligned image patches, wherein the first aligned image patches comprisean extracted portion of first aligned images, wherein the first alignedimages comprise a first image and a second image that have been aligned,wherein the first image comprises a first FOV of the first biologicalsample that has been stained or processed with a first biological marker(which may include, without limitation, a first stain, a firstchromogen, or other suitable compound that is useful for characterizingthe first biological sample, or the like), wherein the second imagecomprises a second FOV of the first biological sample that has beenstained or processed with a second biological marker (which may include,without limitation, a second stain, a second chromogen, or othersuitable compound that is useful for characterizing the first biologicalsample, or the like), wherein the first aligned image patches compriselabeling of instances of features of interest as shown in the extractedportion of the first aligned images; and may utilize the AI system totrain a second AI model (“Model G”) to identify instances of features ofinterest in the first biological sample, based at least in part on thefirst instance classification of features of interest generated by ModelG*.

Alternatively, or additionally, the computing system may generate groundtruth for developing accurate AI models for biological imageinterpretation, based at least in part on images of a first biologicalsample depicting sequential staining or processing of the firstbiological sample.

In another aspect, the computing system may receive a first image of afirst biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample that has been stained orprocessed with a first biological marker (which may include, withoutlimitation, a first stain, a first chromogen, or other suitable compoundthat is useful for characterizing the first biological sample, or thelike); may receive a second image of the first biological sample, thesecond image comprising a second FOV of the first biological sample thathas been stained or processed with at least a second biological marker(which may include, without limitation, a second stain, a secondchromogen, or other suitable compound that is useful for characterizingthe first biological sample, or the like); and may align the first imagewith the second image to create first aligned images, by aligning one ormore features of interest in the first biological sample as depicted inthe first image with the same one or more features of interest in thefirst biological sample as depicted in the second image.

The computing system may create first aligned image patches from thefirst aligned images, by extracting a portion of the first alignedimages, the portion of the first aligned images comprising a first patchcorresponding to the extracted portion of the first image and a secondpatch corresponding to the extracted portion of the second image; mayutilize an AI system to train a first AI model (“Model F”) to generate athird patch comprising a virtual stain of the first aligned imagepatches, based at least in part on the first patch and the second patch,the virtual stain simulating staining or processing by at least thesecond biological marker of features of interest in the first biologicalsample as shown in the second patch; and may utilize the AI system totrain a second model (“Model G*”) to identify or classify firstinstances of features of interest in the first biological sample, basedat least in part on the third patch and based at least in part onresults from an external instance classification process or a region ofinterest detection process.

According to some embodiments, the first image may comprise highlightingor identification of first features of interest in the first biologicalsample by the first biological marker that had been applied to the firstbiological sample. In some cases, the third patch may comprise one ofhighlighting or identification of the first features of interest by thefirst biological marker and highlighting of second features of interestin the first biological sample by the virtual stain that simulates thesecond biological marker having been applied to the first biologicalsample or highlighting or identification of the first features ofinterest by the first biological marker and highlighting oridentification of first features of interest by the virtual stain, thesecond features of interest being different from the first features ofinterest. In some instances, the second biological marker may be one ofthe same as the first biological marker but used to stain or process thesecond features of interest or different from the first biologicalmarker.

In some embodiments, utilizing the AI system to train Model F maycomprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, using the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may include,but is not limited to, one of a mean squared error loss function, a meansquared logarithmic error loss function, a mean absolute error lossfunction, a Huber loss function, or a weighted sum of squareddifferences loss function, and/or the like.

Alternatively, utilizing the AI system to train Model F may comprise:receiving, with the AI system, the first patch; receiving, with the AIsystem, the second patch; generating, with a second model of the AIsystem, an image of the virtual stain; adding the generated image of thevirtual stain to the received first patch to produce a predictedvirtually stained image patch; determining a first loss value betweenthe predicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, using the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some embodiments, the second model mayinclude, without limitation, at least one of a convolutional neuralnetwork (“CNN”), a U-Net, an artificial neural network (“ANN”), aresidual neural network (“ResNet”), an encode/decode CNN, anencode/decode U-Net, an encode/decode ANN, or an encode/decode ResNet,and/or the like.

Alternatively, or additionally, the computing system may receive a firstimage of a first biological sample, the first image comprising a firstfield of view (“FOV”) of the first biological sample; and may identify,using a first model (“Model G*”) that is generated or updated by atrained AI system, first instances of features of interest in the firstbiological sample, based at least in part on the first image and basedat least in part on training of Model G* using at least a first patchcomprising a virtual stain of first aligned image patches, the firstpatch being generated by a second AI model (Model F) that is generatedor updated by the trained AI system by using a second patch. In somecases, the first aligned image patches may comprise an extracted portionof first aligned images. The first aligned images may comprise a secondimage and a third image that have been aligned. The second image maycomprise a second FOV of a second biological sample that is differentfrom the first biological sample that has been stained or processed witha first biological marker (which may include, without limitation, afirst stain, a first chromogen, or other suitable compound that isuseful for characterizing the first biological sample, or the like). Thesecond patch may comprise the extracted portion of the second image. Thethird image may comprise a third FOV of the second biological samplethat has been stained or processed with at least a second biologicalmarker (which may include, without limitation, a second stain, a secondchromogen, or other suitable compound that is useful for characterizingthe second biological sample, or the like).

In order to overcome the problems and limitations with conventionaltechniques (as noted above), the present teachings provide for improvedannotation analysis and for generating cell-perfect annotations as aninput for training of an automated image analysis architecture. This canbe accomplished by leveraging differential tissue imagings such ascomparing a baseline, control, or uncontaminated image of a tissuefollowed by a second image of the same tissue, but now with the objectsof interest stained or processed to be visible or identifiable. Withgood image alignment, such processes can generate tens of thousands (ormore) of accurate and informative cell-level annotations, withoutrequiring the intervention of a human observer, such as a technician orpathologist, or the like. Hence, the potential to generate high qualityand comprehensive training data for a deep learning or othercomputational methods is immense and may greatly improve the performanceof automated and semi-automated image analysis where the ground truthmay be established directly from biological data and reduces or removeshuman observer-based errors and limitations. In addition, the speed ofannotation may be vastly improved compared to the time it would take apathologist to annotate similar datasets.

There are many commercial applications for this technology, such asdeveloping image analysis techniques suitable for use with PharmDx kits.PD-L1 is a non-limiting example where a computational image analysismethod may be used to identify and/or designate lymphocytescomplimenting and extending the assay. For instance, AI models developedbased on the methods described herein can be used to develop digitalscoring of PD-L1 immunohistochemistry (“IHC”) products (including, butnot limited to, Agilent PD-L1 IHC 22C3 family of products, or otherPD-L1 IHC family of products, or the like). For example, PD-L1 IHC 22C3pharmDx interpretation manual for NSCLC may require calculation of aTumor Proportion Score (“TPS”), which may be calculated as thepercentage of viable tumor cells showing partial or complete membranestaining relative to all viable tumor cells present in the sample(positive and negative) of the number of PD-L1 staining cells (includingtumor cells, lymphocytes, macrophages, and/or the like) divided by thetotal number of viable tumor cells, multiplied by 100. Infiltratingimmune cells, normal cells, and necrotic cells may be excluded from theTPS calculation. AI models trained by the methods presented hereinenable exclusion of various immune cell types such as T-cells, B-cells,and macrophages, or the like. Following this exclusion, cells may befurther classified as PD-L1 positive, PD-L1 negative cells, viable tumorcells, or non-tumor cells, or the like, and accurate TPS scores may becalculated. Methods described herein may also be relevant for thecalculation of a Combined Positive Score (“CPS”), or the like. Anothernon-limiting example is identification of p40-positive tumor cells todifferentiate squamous cell carcinomas and adenocarcinomas for non-smallcell lung cancer (“NSCLC”) specimens.

Various embodiments of the present teachings provide a two-step IHCstaining or processing with different biological markers (which mayinclude, without limitation, stains, chromogens, or other suitablecompounds that are useful for characterizing biological samples, or thelike). Biological markers may be visualized using brightfield orfluorescent imaging. In various embodiments described herein, two setsof methods - namely, (a) sequential imaging of (sequentially stained orprocessed) biological samples for generating training data fordeveloping deep learning based models for image analysis, for cellclassification, and/or for features of interest identification ofbiological samples and (b) sequential imaging of (sequentially stainedor processed) biological samples for generating training data forvirtual staining of biological samples - may be used to generate thenecessary data.

These and other aspects of the sequential imaging of biological samplefor generating training data for developing deep learning based modelsfor image analysis, for cell classification, for feature of interestidentification, and/or for virtual staining of biological samples aredescribed in greater detail with respect to the figures. Although thevarious embodiments are described in terms of digital pathologyimplementations, the various embodiments are not so limited, and may beapplicable to live cell imaging, or the like.

Various embodiments described herein, while embodying (in some cases)software products, computer-performed methods, and/or computer systems,represent tangible, concrete improvements to existing technologicalareas, including, without limitation, annotation collection technology,annotation data collection technology, autonomous annotation technology,deep learning technology for autonomous annotation, and/or the like. Inother aspects, certain embodiments, can improve the functioning of userequipment or systems themselves (e.g., annotation collection system,annotation data collection system, autonomous annotation system, deeplearning system for autonomous annotation, etc.), for example, byreceiving, with a computing system, a first image of a first biologicalsample, the first image comprising a first field of view (“FOV”) of thefirst biological sample that has been stained with a first stain;receiving, with the computing system, a second image of the firstbiological sample, the second image comprising a second FOV of the firstbiological sample that has been stained with a second stain;autonomously creating, with the computing system, a first set of imagepatches based on the first image and the second image, by extracting aportion of the first image and extracting a corresponding portion of thesecond image, the first set of image patches comprising a first patchcorresponding to the extracted portion of the first image and a secondpatch corresponding to the extracted portion of the second image;training, using an artificial intelligence (“AI”) system, a first model(“Model G*”) to generate first instance classification of features ofinterest (“Ground Truth”) in the first biological sample, based at leastin part on the first aligned image patches and the labeling of instancesof features of interest contained in the first patch; training, usingthe AI system, a second AI model (“Model G”) to identify instances offeatures of interest in the first biological sample, based at least inpart on the first patch and the first instance classification offeatures of interest generated by Model G*; and/or the like.

In particular, to the extent any abstract concepts are present in thevarious embodiments, those concepts can be implemented as describedherein by devices, software, systems, and methods that involve specificnovel functionality (e.g., steps or operations), such as, training,using an AI system, a (AI) model (referred to herein as “Model G*”)based on sequentially stained biological samples to generate GroundTruth, which can be used to train another AI model (referred to hereinas “Model G”) to identify instances of features of interest inbiological samples (i.e., to score images of biological samples, or thelike); and, in some cases, to train, using the AI system, an AI model(referred to herein as “Model F”) to generate virtual staining ofbiological samples, which can be used to train Model G* to identifyinstances of features of interest in biological samples (i.e., to scoreimages of biological samples, or the like); and/or the like, to name afew examples, that extend beyond mere conventional computer processingoperations. These functionalities can produce tangible results outsideof the implementing computer system, including, merely by way ofexample, optimize sample annotation techniques and systems to improveprecision and accuracy in identifying features of interest in biologicalsamples (that in some instances are difficult or impossible for humansto distinguish or identify) that, in some cases, obviates the need toapply one or more additional stains for sequential staining (byimplementing virtual staining, as described according to the variousembodiments), and/or the like, at least some of which may be observed ormeasured by users and/or service providers.

A. Summary for Tissue Staining Methodology

The present disclosure provides detectable reagents capable of servingas substrates of an enzyme with peroxidase activity, and describes theirutility for detecting molecular targets in samples. The presentdisclosure also provides methods for detecting multiple target moleculesin biological samples comprising cells.

One general aspect includes a method for detecting multiple targetmolecules in a biological sample comprising cells. The method includescontacting the biological sample with one or more first reagents whichgenerate a detectable signal in cells may include a first targetmolecule, contacting the biological sample with one or more secondreagents which are capable of generating a detectable signal in cells,comprising a second target molecule under conditions in which the one ormore second reagents do not generate a detectable signal. The method mayalso include detecting the signal generated by the one or more firstreagents. The method may also include creating conditions in which theone or more second reagents generate a signal in cells comprising thesecond molecule. The method also includes detecting the signal generatedby the one or more second reagents.

The method may include: obtaining a first digital image of the signalgenerated by the one or more first reagents, obtaining a second digitalimage of the signal generated by the one or more first reagents and thesignal generated by the one or more second reagents, and copying a maskof the second digital image to the first digital image.

The biological sample may include one of a human tissue sample, ananimal tissue sample, or a plant tissue sample. The target molecules maybe indicative of at least one of normal cells, cell type, abnormalcells, damaged cells, cancer cells, tumors, subcellular structures, amarker indicative of or associated with a disease, disorder or healthcondition, or organ structures. The target molecules may be selectedfrom the group consisting of nuclear proteins, cytoplasmic proteins,membrane proteins, nuclear antigens, cytoplasmic antigens, membraneantigens and nucleic acids. The target molecules may be nucleic acids orpolypeptides.

The first reagents may include a first primary antibody against a firsttarget molecule, an HRP coupled polymer that binds to the first primaryantibody and a chromogen which is an HRP substrate.

The second reagents may include a second primary antibody against asecond target molecule, an HRP coupled polymer that binds to the secondprimary antibody, a fluorescein-coupled long single chained polymer, anantibody against FITC that binds to the fluorescein-coupled long singlechained polymer and is coupled to HRP and a chromogen which is an HRPsubstrate.

The method may include using a digital image of the signals detected inthe biological sample to train a neural network.

A neural network trained using the methods is also contemplated.

Another aspect includes a method for generating cell-level annotationsfrom a biological sample comprising cells. The method includes a)exposing the biological sample to a first ligand that recognizes a firstantigen thereby forming a first ligand antigen complex; b) exposing thefirst ligand antigen complex to a first labeling reagent binding to thefirst ligand, the first labeling reagent forming a first detectablereagent, where the first detectable reagent is precipitated around thefirst antigen and visible in brightfield; c) exposing the biologicalsample to a second ligand that recognizes a second antigen therebyforming a second ligand antigen complex; d) exposing the second ligandantigen complex to a second labeling reagent binding to the secondligand, the second labeling reagent may include a substrate not visiblein brightfield, where the substrate is precipitated around the secondantigen; e) obtaining a first image of the biological sample inbrightfield to visualize the first chromogen precipitated in thebiological sample; f) exposing the biological sample to a third labelingreagent that recognizes the substrate, thereby forming a third ligandantigen complex, the third labeling reagent forming a second detectablereagent, where the second detectable reagent is precipitated around thesecond antigen; g) obtaining a second image of the biological sample inbrightfield with the second detectable reagent precipitated in thebiological sample; h) creating a mask from the second image; i) applyingthe mask to the first image so as to obtain an image of the biologicalsample annotated with the second antigen.

The biological sample may include one of a human tissue sample, ananimal tissue sample, or a plant tissue sample, where the objects ofinterest may include at least one of normal cells, abnormal cells,damaged cells, cancer cells, tumors, subcellular structures, or organstructures.

In this aspect, the method may include, prior to step a), applying atarget retrieval buffer and protein blocking solution to the biologicalsample, where the first and second antigens are exposed for thesubsequent steps and endogenous peroxidases are inactivated.

The first and third labeling reagents may include an enzyme that acts ona detectable reagent substrate to form the first and second detectablereagents, respectively.

The first and second antigens may be non-nuclear proteins.

The method may include, following step b), denaturing the first ligandsto retrieve the first antigens available.

The method may include, following step d), i) counterstaining cellnuclei of the biological sample, and ii) dehydrating and mounting thesample on a slide.

The method may include, following step e), i) removing mounting mediumfrom the slide, and ii) rehydrating the biological sample.

The method may include, following step f), i) counterstaining cellnuclei of the biological sample, and ii) dehydrating and mounting thesample on a slide.

In some aspects, the first ligand may include ananti-lymphocyte-specific antigen antibody (primary antibody), the firstlabeling reagent may include a horseradish peroxidase (HRP)-coupledpolymer capable of binding to the primary antibody, the first detectablereagent may include a chromogen may include an HRP substrate may includeHRP magenta or 3,3′-diaminobenzidine tetrahydrochloride (DAB), thesecond antigen may include PD-L1 the second ligand may includeanti-PD-L1 antibodies, the second labeling reagent may include anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, thesubstrate may include fluorescein-coupled long single-chained polymer(fer-4-flu linker; see U.S. Pat. Application Publication No.2016/0122800, incorporated by reference in its entirety), third labelingreagent may include an anti-FITC antibody coupled to HRP, and the seconddetectable reagent may include a chromogen may include HRP magenta orDAB.

Fer-4-flu from U.S. Pat. Application Publication No. 2016/0122800 isshown in FIG. 22 .

In some aspects, the first labeling reagent may include a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent may include a chromogen mayinclude an HRP substrate may include HRP magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB), the second antigen mayinclude p40 the second ligand may include anti-p40 antibodies, thesecond labeling reagent may include an HRP-coupled polymer capable ofbinding to anti-p40 antibodies, the substrate may includefluorescein-coupled long single-chained polymer (fer-4-flu linker),third labeling reagent an anti-FITC antibody coupled to HRP, and thesecond detectable reagent may include a chromogen may include HRPmagenta or dab. It will be appreciated that the antigens may be stainedin any order. For example, the p40 antigen may be stained first,followed by the lymphocyte-specific antigen. Alternatively, thelymphocyte-specific antigen may be stained first, followed by the p40antigen.

A counterstaining agent may be hematoxylin.

In some aspects, the method further includes using a digital image ofthe signals detected in the biological sample to train a neural network.

A neural network trained using the methods is also contemplated.

An annotated image obtained by the methods is also contemplated.

A further aspect includes a method for detecting multiple targetmolecules in a biological sample comprising cells. The method alsoincludes contacting the biological sample with one or more firstreagents which generate a detectable signal in cells may include a firsttarget molecule. The method also includes contacting the biologicalsample with one or more second reagents which generate a detectablesignal in cells; it may include a second target molecule, where thedetectable signal generated by the one or more second reagents isremovable. The method also includes detecting the signal generated bythe one or more first reagents and the signal generated by the one ormore second reagents. The method also includes creating conditions inwhich the signal generated by the one or more second reagents isremoved. The method also includes detecting the signal generated by theone or more first reagents.

Implementations may include one or more of the following features. Themethod may include: obtaining a first digital image of the signalgenerated by the one or more first reagents and the signal generated bythe one or more second reagents, obtaining a second digital image of thesignal generated by the one or more first reagents, and copying a maskof the first digital image to the second digital image.

The biological sample may comprise one of a human tissue sample, ananimal tissue sample, or a plant tissue sample, wherein the objects ofinterest comprise at least one of normal cells, abnormal cells, damagedcells, cancer cells, tumors, subcellular structures, or organstructures.

The target molecules are indicative of at least one of normal cells,cell type, abnormal cells, damaged cells, cancer cells, tumors,subcellular structures, a marker indicative of or associated with adisease, disorder or health condition, or organ structures.

The target molecules may include nuclear proteins, cytoplasmic proteins,membrane proteins, nuclear antigens, cytoplasmic antigens, membraneantigens and nucleic acids. The target molecules may be nucleic acids orpolypeptides.

The first reagents may include a first primary antibody against a firsttarget molecule, an HRP coupled polymer that binds to the first primaryantibody and a chromogen which is an HRP substrate.

The second reagents may include a second primary antibody against asecond target molecule, an HRP coupled polymer that binds to the secondprimary antibody, and amino ethyl carbazole.

The method may include using a digital image of the signals detected inthe biological sample to train a neural network. A neural networktrained using the method is also contemplated.

Another aspect includes a method for generating cell-level annotationsfrom a biological sample comprising cells. The method includes a)exposing the biological sample to a first ligand that recognizes a firstantigen thereby forming a first ligand antigen complex. The method alsoincludes b) exposing the first ligand antigen complex to a firstlabeling reagent binding to the first ligand, the first labeling reagentforming a first detectable reagent, where the first detectable reagentis precipitated around the first antigen. The method also includes c)exposing the biological sample to a second ligand that recognizes asecond antigen thereby forming a second ligand antigen complex. Themethod also includes d) exposing the second ligand antigen complex to asecond labeling reagent binding to the second ligand, the secondlabeling reagent forming a second detectable reagent, where the seconddetectable reagent is precipitated around the second antigen. The methodalso includes e) obtaining a first image of the biological sample withthe first and second detectable reagents precipitated in the biologicalsample. The method also includes f) incubating the tissue sample with anagent which dissolves the second detectable reagent. The method alsoincludes g) obtaining a second image of the biological sample with thefirst detectable reagent precipitated in the biological sample. Themethod also includes h) creating a mask from the first image. The methodalso includes i) applying the mask to the second image so as to obtainan annotated image of the biological sample with the second antigen.

Implementations may include one or more of the following features. Thefirst biological sample may include one of a human tissue sample, ananimal tissue sample, or a plant tissue sample, where the objects ofinterest may include at least one of normal cells, abnormal cells,damaged cells, cancer cells, tumors, subcellular structures, or organstructures.

The method may include, prior to step a), applying a target retrievalbuffer and protein blocking solution to the biological sample, where thefirst and second antigens are exposed for the subsequent steps andendogenous peroxidases are inactivated.

The first and second labeling reagents may include an enzyme that actson a detectable reagent substrate to form the first and seconddetectable reagents, respectively.

The first and second antigens may be non-nuclear proteins.

The method may include, following step b), denaturing the first ligandsto retrieve the first antigens available.

The method may include, following step d), i) counterstaining cellnuclei of the biological sample, and ii) dehydrating and mounting thesample on a slide.

The method may include, following step e), i) removing mounting mediumfrom the slide, and ii) rehydrating the biological sample.

The method may include, following step f), dehydrating and mounting thesample on a slide.

The first antigen may include a lymphocyte-specific antigen, the firstligand may include an anti-lymphocyte-specific antigen antibody (primaryantibody), the first labeling reagent may include a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent may include an HRP substrate mayinclude HRP magenta or 3,3′-diaminobenzidine tetrahydrochloride (DAB),the second antigen may include PD-L1 the second ligand may includeanti-PD-L1 antibodies, the second labeling reagent may include anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, thesecond detectable reagent may include amino ethyl carbazole (AEC), andthe agent which dissolves the second detectable reagent is alcohol oracetone.

The first labeling reagent may include a horseradish peroxidase(HRP)-coupled polymer capable of binding to the primary antibody, thefirst detectable reagent may include an HRP substrate may include HRPmagenta or 3,3′-diaminobenzidine tetrahydrochloride (dab), the secondantigen may include p40 the second ligand may include anti-p40antibodies, the second labeling reagent may include an HRP-coupledpolymer capable of binding to anti-p40 antibodies, the second detectablereagent may include amino ethyl carbazole (AEC), and the agent whichdissolves the second detectable reagent is alcohol or acetone. It willbe appreciated that the antigens may be stained in any order. Forexample, the p40 antigen may be stained first, followed by thelymphocyte-specific antigen. Alternatively, the lymphocyte-specificantigen may be stained first, followed by the p40 antigen.

A counterstaining agent can be hematoxylin.

The method may include using a digital image of the signals detected inthe biological sample to train a neural network.

An annotated image obtained by the method is also contemplated. A neuralnetwork trained using the method is also contemplated.

Another aspect includes a method for detecting multiple target moleculesin a biological sample comprising cells. The method includes contactingthe biological sample with one or more first reagents which generate afirst detectable signal in cells may include a first target molecule,where said first detectable signal is detectable using a first detectionmethod. The method also includes contacting the biological sample withone or more second reagents which generate a second detectable signaland may include a second target molecule, where the second detectablesignal is detectable using a second detection method and issubstantially undetectable using the first detection method and wherethe first detectable signal is substantially undetectable using thesecond detection method. The method also includes detecting the signalgenerated by the one or more first reagents using the first detectionmethod. The method also includes detecting the signal generated by theone or more second reagents using the second detection method.

Implementations may include one or more of the following features. Thebiological sample may include one of a human tissue sample, an animaltissue sample, or a plant tissue sample, where the objects of interestmay include at least one of normal cells, abnormal cells, damaged cells,cancer cells, tumors, subcellular structures, or organ structures.

The target molecules may be indicative of at least one of normal cells,cell type, abnormal cells, damaged cells, cancer cells, tumors,subcellular structures, a marker indicative of or associated with adisease, disorder or health condition, or organ structures.

The target molecules are selected from the group consisting of nuclearproteins, cytoplasmic proteins, membrane proteins, nuclear antigens,cytoplasmic antigens, membrane antigens and nucleic acids. The targetmolecules may include nucleic acids or polypeptides.

The first reagents may include a first primary antibody against a firsttarget molecule, an HRP coupled polymer that binds to the first primaryantibody and a chromogen which is an HRP substrate.

The second reagents may include a second primary antibody against asecond target molecule, an HRP coupled polymer that binds to the secondprimary antibody, and a rhodamine based fluorescent compound coupled toa long single chained polymer.

The method may include using a digital image of the signals detected inthe biological sample to train a neural network. A neural networktrained using the method is also contemplated.

A further aspect includes a method for generating cell-level annotationsfrom a biological sample. The method includes a) exposing the biologicalsample to a first ligand that recognizes a first antigen thereby forminga first ligand antigen complex. The method also includes b) exposing thefirst ligand antigen complex to a first labeling reagent binding to thefirst ligand, the first labeling reagent forming a first detectablereagent, where the first detectable reagent is visible in brightfield;where the first detectable reagent is precipitated around the firstantigen. The method also includes c) exposing the biological sample to asecond ligand that recognizes a second antigen thereby forming a secondligand antigen complex. The method also includes d) exposing the secondligand antigen complex to a second labeling reagent binding to thesecond ligand, the second labeling reagent forming a second detectablereagent, where the second detectable reagent is visible in fluorescence;where the second detectable reagent is precipitated around the secondantigen. The method also includes e) obtaining a first brightfield imageof the biological sample with the first detectable reagent precipitatedin the biological sample. The method also includes f) obtaining a secondfluorescent image of the biological sample with the second detectablereagent precipitated in the biological sample. The method also includesg) creating a mask from the second image. The method also includes h)applying the mask to the first image so as to obtain an annotated imageof the tissue sample with the second marker.

Implementations may include one or more of the following features. Thefirst biological sample may include one of a human tissue sample, ananimal tissue sample, or a plant tissue sample, where the objects ofinterest may include at least one of normal cells, abnormal cells,damaged cells, cancer cells, tumors, subcellular structures, or organstructures.

The method may include, prior to step a), applying a target retrievalbuffer and protein blocking solution to the biological sample, where thefirst and second antigens are exposed for the subsequent steps andendogenous peroxidases are inactivated.

The first and second labeling reagents may include an enzyme that actson a detectable reagent substrate to form the first and seconddetectable reagents, respectively.

The first and second antigens may be non-nuclear proteins.

The method may include, following step b), denaturing the first ligandsto retrieve the first antigens available.

The method may include, following step d), i) counterstaining cellnuclei of the biological sample, and ii) dehydrating and mounting thesample on a slide.

The first labeling reagent may include a horseradish peroxidase(HRP)-coupled polymer capable of binding to the primary antibody, thefirst detectable reagent may include an HRP substrate may include HRPmagenta or 3,3′-diaminobenzidine tetrahydrochloride (dab), the secondantigen may include PD-L1 the second ligand may include anti-PD-L1antibodies, the second labeling reagent may include an HRP-coupledpolymer capable of binding to anti-PD-L1 antibodies, and the seconddetectable reagent may include a rhodamine-based fluorescent compoundcoupled to a long single-chain polymer.

The first antigen may include a lymphocyte-specific antigen, the firstligand may include an anti-lymphocyte-specific antigen antibody (primaryantibody), the first labeling reagent may include a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent may include an HRP substrate mayinclude HRP magenta or 3,3′-diaminobenzidine tetrahydrochloride (DAB),the second antigen may include p40 the second ligand may includeanti-p40 antibodies, the second labeling reagent may include anHRP-coupled polymer capable of binding to anti-p40 antibodies, and thesecond detectable reagent may include rhodamine-based fluorescentcompound coupled to a long single-chain polymer. It will be appreciatedthat the antigens may be stained in any order. For example, the p40antigen may be stained first, followed by the lymphocyte-specificantigen. Alternatively, the lymphocyte-specific antigen may be stainedfirst, followed by the p40 antigen.

A counterstaining agent may be hematoxylin. An annotated image obtainedby the methods is also contemplated. A neural network trained using themethod is also contemplated.

One aspect includes a compound of Formula I:

where X is —COOR^(x), or —CH₂COOR^(x), or —CONR^(x)R^(xx), or—CH₂CONR^(x)R^(xx); where Y is ═O, ═NR^(Y), or ═N⁺R^(Y) _(R) ^(YY) ;wherein R¹, R², R³, R⁴, R⁵, R⁶, R⁷, R⁸, R⁹, R¹⁰, R^(X), R^(xx), R^(Z),R^(Y), and R^(YY) are independently selected from hydrogen and asubstituent having less than 40 atoms;

L is a linker comprising a linear chain of 5 to 29 consecutivelyconnected atoms; and PS is a peroxidase substrate moiety.

In some aspects, R¹ is selected from hydrogen, R¹¹, (C1-C20) alkyl orheteroalkyl optionally substituted with one or more of the same ordifferent R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally substitutedwith one or more of the same or different R¹³ or suitable R¹⁴ groups and(C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with oneor more of the same or different R¹³ or suitable R¹⁴ groups, oralternatively, R¹ may be taken together with R² to form part of a benzo,naptho or polycyclic aryleno group which is optionally substituted withone or more of the same or different R¹³ or suitable R¹⁴ groups;

-   R² is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or    alternatively, R² may be taken together with R¹, to form part of a    benzo, naptho or polycyclic aryleno group which is optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups;-   R^(X), when present, is selected from hydrogen, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups;-   R^(xx), when present, is selected from (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups;-   R³ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups;-   R⁴ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively, when Y is -N+R^(Y)R^(YY), R⁴ may be taken together    with R^(yy) to form a 5- or 6-membered ring which is optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups;-   R^(yy), when present, is selected from (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively R^(yy) may be taken together with R⁴ to form a 5- or    6-membered ring which is optionally substituted with one or more of    the same or different R¹³ or suitable R¹⁴ groups;-   R^(Y), when present, is selected from hydrogen, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups, or, alternatively, R^(Y) may be taken    together with R⁵ to form a 5- or 6-membered ring optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups;-   R^(Z), when present, is selected from hydrogen, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups;-   R⁵ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively, R⁵ may be taken together with R⁶ to form part of a    benzo, naptho or polycyclic aryleno group which is optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups, or alternatively, when Y is -N+R^(Y)R^(YY), R⁵    may be taken together with R^(y) to form a 5- or 6-membered ring    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups;-   R⁶ is selected from hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl    optionally substituted with one or more of the same or different R¹⁴    groups, (C5-C20) aryl or heteroaryl optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups and    (C6-C40) arylalkyl or heteroaryl alkyl optionally substituted with    one or more of the same or different R¹³ or suitable R¹⁴ groups, or,    alternatively, R⁶ together with R⁵ may form part of a benzo, naptho    or polycyclic aryleno group which is optionally substituted with one    or more of the same or different R¹³ or suitable R¹⁴ groups;-   R⁷, R⁸ and R⁹ are each, independently of one another, selected from    hydrogen, R¹¹, (C1-C20) alkyl or heteroalkyl optionally substituted    with one or more of the same or different R¹⁴ groups, (C5-C20) aryl    or heteroaryl optionally substituted with one or more of the same or    different R¹³ or suitable R¹⁴ groups and (C6-C40) arylalkyl or    heteroaryl alkyl optionally substituted with one or more of the same    or different R¹³ or suitable R¹⁴ groups;-   R¹⁰ is selected from selected from hydrogen, R¹¹, (C1-C20) alkyl or    heteroalkyl optionally substituted with one or more of the same or    different R¹⁴ groups, (C5-C20) aryl or heteroaryl optionally    substituted with one or more of the same or different R¹³ or    suitable R¹⁴ groups and (C6-C40) arylalkyl or heteroaryl alkyl    optionally substituted with one or more of the same or different R¹³    or suitable R¹⁴ groups, halo, haloalkyl, —OR¹², —SR¹², —SOR¹²,    -SO₂R¹², and nitrile;-   R¹¹ is selected from —NR¹⁵R¹⁵, —OR¹⁶, —SR¹⁶, halo, haloalkyl, —CN,    —NC, —OCN, —SCN, —NO, —NO₂, —N₃, —S(O)R¹⁶, —S(O)₂R¹⁶, —S(O)₂OR¹⁶,    —S(O)NR¹⁵R¹⁵, —S(O)₂NR¹⁵R¹⁵ —OS(O)R¹⁶, —OS(O)₂R¹⁶, —OS(O)₂NR¹⁵R¹⁵,    —OP(O)₂R¹⁶, —OP(O)₃R¹⁶R¹⁶, —P(O)₃R¹⁶R¹⁶, —C(O)R¹⁶, —C(O)OR¹⁶,    —C(O)NR¹⁵R¹⁵, —C(NH)NR¹⁵R¹⁵, —OC(O)R¹⁶, —OC(O)OR¹⁶, —OC(O)NR¹⁵R¹⁵    and —OC(NH)NR¹⁵R¹⁵;-   R¹² is selected from (C1-C20) alkyls or heteroalkyls optionally    substituted with lipophilic substituents, (C5-C20) aryls or    heteroaryls optionally substituted with lipophilic substituents and    (C2-C26) arylalkyl or heteroarylalkyls optionally substituted with    lipophilic substituents;-   R¹³ is selected from hydrogen, (C1-C8) alkyl or heteroalkyl,    (C5-C20) aryl or heteroaryl and (C6-C28) arylalkyl or    heteroarylalkyl;-   R¹⁴ is selected from —NR¹⁵R¹⁵, ═O, —OR¹⁶, ═S, —SR¹⁶, —NR¹⁶, —NOR¹⁶,    halo, haloalkyl, —CN, —NC, —OCN, —SCN, —NO, —NO₂, ═N₂, —N₃,    —S(O)R¹⁶, —S(O)₂R¹⁶, —S(O)₂OR¹⁶, —S(O)NR¹⁵R¹⁵, —OS(O)₂R¹⁶,    —OS(O)₂NR¹⁵R¹⁵,—OS(O)₂OR¹⁶, —OS(O)₂NR¹⁵R¹⁵, —C(O)R ¹⁶, —C(O)OR¹⁶,    —C(O)NR¹⁵R¹⁵, —C(NH)NR¹⁵R¹⁵, —OC(O)R¹⁶, —OC(O)OR¹⁶, —OC(O)NR¹⁵R¹⁵    and —OC(NH)NR¹⁵R¹⁵; each R¹⁵ is independently hydrogen or R¹⁶, or    alternatively, each R¹⁵ is taken together with the nitrogen atom to    which it is bonded to form a 5- to 8-membered saturated or    unsaturated ring which may optionally include one or more of the    same or different additional heteroatoms and which may optionally be    substituted with one or more of the same or different R¹³ or R¹⁶    groups; each R¹⁶ is independently R¹³ or R¹³ substituted with one or    more of the same or different R¹³ or R¹⁷ groups; and each R¹⁷ is    selected from —NR¹³R¹³, —OR¹³, ═S, —SR¹³, ═NR¹³, ═NOR¹³, halo,    haloalkyl, —CN, —NC, —OCN, —SCN, —NO, —NO₂, ═N₂, —N₃, —S(O)R¹³,    —S(O)₂R¹³, —S(O)₂OR¹³, —S(O)NR¹³R¹³, —S(O)₂NT¹³R¹³, —OS(O)R¹³,    —OS(O)₂R¹³, —OS(O)₂NR¹³R¹³, —OS(O)₂OR¹⁶, —OS(O)₂NR¹³R¹³, —C(O)OR¹³,    —C(O)OR¹³, —C(O)NR¹³R¹³, —C(NH)NR¹⁵R¹³, —OC(O)R¹³, —OC(O)OR¹³,    —OC(O)NR¹³R¹³ and —OC(NH)NR¹³R¹³.

In some aspects, the compound comprises a chromogenic moiety selectedfrom the group consisting of rhodamine, rhodamine derivatives,fluorescein, fluorescein derivatives in which X is COOH, CH₂COOH,—CONH₂, —CH₂CONH₂, and salts of the foregoing.

In some aspects, the compound comprises a chromogenic moiety selectedfrom the group consisting of rhodamine, rhodamine 6G,tetramethylrhodamine, rhodamine B, rhodamine 101, rhodamine 110,fluorescein, O-carboxymethyl fluorescein, derivatives of the foregoingin which X is —COOH, —CH₂COOH, —CONH₂, —CH₂CONH₂, and salts of theforegoing.

In some aspects, the peroxidase substrate moiety has the followingformula:

wherein R₂₁ is —H, R₂₂ is —H, —O—X, or —N(X)₂; R₂₃ is —OH; R₂₄ is —H,—O—X, or —N(X)₂; R₂₅ is —H, —O—X, or —N(X)₂; R₂₆ is CO, X is H, alkyl oraryl; wherein PS is linked to L through R²⁶. In some aspects, R₂₃ is—OH, and R₂₄ is —H. In some aspects, either R₂₁ or R₂₅ is —OH, R₂₂ andR₂₄ are —H, and R₂₃ is —OH or —NH₂.

In some aspects, the peroxidase substrate moiety is a residue of ferulicacid, cinnamic acid, caffeic acid, sinapinic acid, 2,4-dihydroxycinnamicacid or 4-hydroxycinnamic acid (coumaric acid).

In some aspects, the peroxidase substrate moiety is a residue of4-hydroxycinnamic acid.

In some aspects, the linker is a compound that comprises (Formula R³⁵):

-   wherein the curved lines denote attachment points to the    compound (CM) and to the peroxidase substrate (PS), and wherein R³⁴    is optional and can be omitted or used as an extension of linker,    wherein R³⁴ is:

-   

-   wherein the curved lines denote attachment point to R³³ and PS,    wherein R³¹ is selected from methyl, ethyl, propyl, OCH₂, CH₂OCH₂,    (CH₂OCH₂)₂, NHCH₂, NH(CH₂)₂, CH₂NHCH₂, cycloalkyl, alkyl-cycloalkyl,    alkyl-cycloalkyl-alkyl, heterocyclyl (such as nitrogen-containing    rings of 4 to 8 atoms), alkyl-heterocyclyl,    alkyl-heterocyclyl-alkyl, and wherein no more than three    consecutively repeating ethyloxy groups, and R³² and R³³ are    independently in each formula elected from NH and O.

In some aspects, the linker is selected from one or two repeat of amoiety of Formula IIIa, IIIb, or IIIc:

with an optional extansion:

which may be inserted between N and PS bond.

In some aspects, Z-L-PS together comprises:

wherein the curved line denotes the attachment point.

In some aspects, the compound has the formula (Formula IV):

In some aspects, the compound has the formula (Formula V):

In some aspects, the compound has the formula (Formula VI):

and their corresponding spiro-derivatives, or salts thereof.

In some aspects, the compound has the formula (formula VII):-.

B. Summary for Sequential Imaging of Biological Samples for GeneratingTraining Data for Developing Deep Learning Based Models for ImageAnalysis, for Cell Classification, for Feature of InterestIdentification, and/or for Virtual Staining of the Biological Sample

In an aspect, a method may comprise receiving, with a computing system,a first image of a first biological sample, the first image comprising afirst field of view (“FOV”) of the first biological sample that has beenstained with a first stain; and receiving, with the computing system, asecond image of the first biological sample, the second image comprisinga second FOV of the first biological sample that has been stained with asecond stain. The method may also comprise autonomously creating, withthe computing system, a first set of image patches based on the firstimage and the second image, by extracting a portion of the first imageand extracting a corresponding portion of the second image, the firstset of image patches comprising a first patch corresponding to theextracted portion of the first image and a second patch corresponding tothe extracted portion of the second image, wherein the first set ofimage patches comprises labeling of instances of features of interest inthe first biological sample that is based at least in part on at leastone of information contained in the first patch, information containedin the second patch, or information contained in one or more externallabeling sources.

The method may further comprise training, using an artificialintelligence (“AI”) system, a first model (“Model G*”) to generate firstinstance classification of features of interest (“Ground Truth”) in thefirst biological sample, based at least in part on the first set ofimage patches and the labeling of instances of features of interestcontained in the first set of image patches; and training, using the AIsystem, a second AI model (“Model G”) to identify instances of featuresof interest in the first biological sample, based at least in part onthe first patch and the first instance classification of features ofinterest generated by Model G*.

In some embodiments, the computing system may comprise one of acomputing system disposed in a work environment, a remote computingsystem disposed external to the work environment and accessible over anetwork, a web server, a web browser, or a cloud computing system,and/or the like. In some instances, the work environment may comprise atleast one of a laboratory, a clinic, a medical facility, a researchfacility, a healthcare facility, or a room.

According to some embodiments, the AI system may comprise at least oneof a machine learning system, a deep learning system, a modelarchitecture, a statistical model-based system, or a deterministicanalysis system, and/or the like. In some instances, the modelarchitecture may comprise at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”), and/or the like. In some cases, the first biological sample maycomprise one of a human tissue sample, an animal tissue sample, a planttissue sample, or an artificially produced tissue sample, and/or thelike, wherein the features of interest may comprise at least one ofnormal cells, abnormal cells, damaged cells, cancer cells, tumors,subcellular structures, or organ structures, and/or the like.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample by the first stainthat had been applied to the first biological sample. Similarly, thesecond image may comprise one of highlighting of second features ofinterest in the first biological sample by the second stain that hadbeen applied to the first biological sample in addition to highlightingof the first features of interest by the first stain or highlighting ofthe second features of interest by the second stain without highlightingof the first features of interest by the first stain, the secondfeatures of interest being different from the first features ofinterest. In some cases, the second stain may be one of the same as thefirst stain but used to stain the second features of interest ordifferent from the first stain, or the like.

Merely by way of example, in some cases, the first stain may comprise atleast one of Hematoxylin, Acridine orange, Bismarck brown, Carmine,Coomassie blue, Cresyl violet, Crystal violet,4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.Likewise, the second stain may comprise at least one of Hematoxylin,Acridine orange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet,Crystal violet, DAPI, Eosin, Ethidium bromide intercalates, Acidfuchsine, Hoechst stain, Iodine, Malachite green, Methyl green,Methylene blue, Neutral red, Nile blue, Nile red, Osmium tetroxide,Propidium Iodide, Rhodamine, Safranine, PD-L1 stain, DAB chromogen,Magenta chromogen, cyanine chromogen, CD3 stain, CD20 stain, CD68 stain,p40 stain, antibody-based stain, or label-free imaging marker, and/orthe like.

According to some embodiments, the first image may comprise one of afirst set of color or brightfield images, a first fluorescence image, afirst phase image, or a first spectral image, and/or the like, whereinthe first set of color or brightfield images may comprise a firstred-filtered (“R”) image, a first green-filtered (“G”) image, and afirst blue-filtered (“B”) image, and/or the like. The first fluorescenceimage may comprise at least one of a first autofluorescence image or afirst labelled fluorescence image having one or more channels withdifferent excitation or emission characteristics, and/or the like. Thefirst spectral image may comprise at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, and/or the like. Similarly, the second image maycomprise one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like, wherein the second set of color or brightfield imagesmay comprise a second R image, a second G image, and a second B image,and/or the like. The second fluorescence image may comprise at least oneof a second autofluorescence image or a second labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may compriseat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, and/or the like.

In some embodiments, the method may further comprise aligning, with thecomputing system, the first image with the second image to create firstset of aligned images, by aligning one or more features of interest inthe first biological sample as depicted in the first image with the sameone or more features of interest in the first biological sample asdepicted in the second image. In some instances, aligning the firstimage with the second image may be one of performed manually usingmanual inputs to the computing system or performed autonomously, whereinautonomous alignment may comprise alignment using at least one ofautomated global alignment techniques or optical flow alignmenttechniques, or the like.

According to some embodiments, the method may further comprisereceiving, with the computing system, a third image of the firstbiological sample, the third image comprising a third FOV of the firstbiological sample that has been stained with a third stain differentfrom each of the first stain and the second stain; and autonomouslyperforming, with the computing system, one of: aligning the third imagewith each of the first image and the second image to create secondaligned images, by aligning one or more features of interest in thefirst biological sample as depicted in the third image with the same oneor more features of interest in the first biological sample as depictedin each of the first image and the second image; or aligning the thirdimage with the first aligned images to create second aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the third image with the same one or more features ofinterest in the first biological sample as depicted in the first alignedimages.

The method may further comprise autonomously creating, with thecomputing system, second aligned image patches from the second alignedimages, by extracting a portion of the second aligned images, theportion of the second aligned images comprising a first patchcorresponding to the extracted portion of the first image, a secondpatch corresponding to the extracted portion of the second image, and athird patch corresponding to the extracted portion of the third image;training the AI system to update Model G* to generate second instanceclassification of features of interest in the first biological sample,based at least in part on the second aligned image patches and thelabeling of instances of features of interest contained in the first setof image patches; and training the AI system to update Model G toidentify instances of features of interest in the first biologicalsample, based at least in part on the first patch and the secondinstance classification of features of interest generated by Model G*.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample by the first stainthat had been applied to the first biological sample. In some cases, thesecond image may comprise one of highlighting of second features ofinterest in the first biological sample by the second stain that hadbeen applied to the first biological sample in addition to highlightingof the first features of interest by the first stain or highlighting ofthe second features of interest by the second stain without highlightingof the first features of interest by the first stain, the secondfeatures of interest being different from the first features ofinterest. In some instances, the third image may comprise one ofhighlighting of third features of interest in the first biologicalsample by the third stain that had been applied to the first biologicalsample in addition to highlighting of the first features of interest bythe first stain and highlighting of the second features of interest bythe second stain, highlighting of the third features of interest by thethird stain in addition to only one of highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain, or highlighting of the thirdfeatures of interest by the third stain without any of highlighting ofthe first features of interest by the first stain or highlighting of thesecond features of interest by the second stain, and/or the like, thethird features of interest being different from each of the firstfeatures of interest and the second features of interest.

According to some embodiments, the first image may comprise one of afirst set of color or brightfield images, a first fluorescence image, afirst phase image, or a first spectral image, and/or the like, whereinthe first set of color or brightfield images may comprise a firstred-filtered (“R”) image, a first green-filtered (“G”) image, and afirst blue-filtered (“B”) image, and/or the like. The first fluorescenceimage may comprise at least one of a first autofluorescence image or afirst labelled fluorescence image having one or more channels withdifferent excitation or emission characteristics, and/or the like. Thefirst spectral image may comprise at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, and/or the like. Likewise, the second image may compriseone of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like, wherein the second set of color or brightfield imagesmay comprise a second R image, a second G image, and a second B image,and/or the like. The second fluorescence image may comprise at least oneof a second autofluorescence image or a second labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may compriseat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, and/or the like. Similarly, thethird image may comprise one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, wherein the third set of color orbrightfield images may comprise a third R image, a third G image, and athird B image, and/or the like. The third fluorescence image maycomprise at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may comprise at least one of a third Raman spectroscopyimage, a third NIR spectroscopy image, a third multispectral image, athird hyperspectral image, or a third full spectral image, and/or thelike.

In some embodiments, the method may further comprise receiving, with thecomputing system, a fourth image, the fourth image comprising one of afourth FOV of the first biological sample different from the first FOVand the second FOV or a fifth FOV of a second biological sample, whereinthe second biological sample is different from the first biologicalsample; and identifying, using Model G, second instances of features ofinterest in the second biological sample, based at least in part on thefourth image and based at least in part on training of Model G using thefirst patch and the first instance classification of features ofinterest generated by Model G*. In some cases, the fourth image maycomprise the fifth FOV and may further comprise only highlighting offirst features of interest in the second biological sample by the firststain that had been applied to the second biological sample. Accordingto some embodiments, the method may further comprise generating, usingModel G, a clinical score, based at least in part on the identifiedsecond instances of features of interest.

In another aspect, a system may comprise a computing system, which maycomprise at least one first processor and a first non-transitorycomputer readable medium communicatively coupled to the at least onefirst processor. The first non-transitory computer readable medium mayhave stored thereon computer software comprising a first set ofinstructions that, when executed by the at least one first processor,causes the computing system to: receive a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample that has been stained with afirst stain; receive a second image of the first biological sample, thesecond image comprising a second FOV of the first biological sample thathas been stained with a second stain; autonomously create a first set ofimage patches based on the first image and the second image, byextracting a portion of the first image and extracting a correspondingportion of the second image, the first set of image patches comprising afirst patch corresponding to the extracted portion of the first imageand a second patch corresponding to the extracted portion of the secondimage, wherein the first set of image patches may comprise labeling ofinstances of features of interest in the first biological sample that isbased at least in part on at least one of information contained in thefirst patch, information contained in the second patch, or informationcontained in one or more external labeling sources; train, using anartificial intelligence (“AI”) system, a first model (“Model G*”) togenerate first instance classification of features of interest (“GroundTruth”) in the first biological sample, based at least in part on thefirst set of image patches and the labeling of instances of features ofinterest contained in the first set of image patches; and train, usingthe AI system, a second AI model (“Model G”) to identify instances offeatures of interest in the first biological sample, based at least inpart on the first patch and the first instance classification offeatures of interest generated by Model G*.

In some embodiments, the computing system may comprise one of acomputing system disposed in a work environment, a remote computingsystem disposed external to the work environment and accessible over anetwork, a web server, a web browser, or a cloud computing system,and/or the like. In some cases, the work environment may comprise atleast one of a laboratory, a clinic, a medical facility, a researchfacility, a healthcare facility, or a room, and/or the like. In someinstances, the AI system may comprise at least one of a machine learningsystem, a deep learning system, a model architecture, a statisticalmodel-based system, or a deterministic analysis system, and/or the like.In some instances, the model architecture may comprise at least one of aneural network, a convolutional neural network (“CNN”), or a fullyconvolutional network (“FCN”), and/or the like.

According to some embodiments, the first biological sample may compriseone of a human tissue sample, an animal tissue sample, a plant tissuesample, or an artificially produced tissue sample, and/or the like. Insome instances, the features of interest may comprise at least one ofnormal cells, abnormal cells, damaged cells, cancer cells, tumors,subcellular structures, or organ structures, and/or the like.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample by the first stainthat had been applied to the first biological sample. In some cases, thesecond image may comprise one of highlighting of second features ofinterest in the first biological sample by the second stain that hadbeen applied to the first biological sample in addition to highlightingof the first features of interest by the first stain or highlighting ofthe second features of interest by the second stain without highlightingof the first features of interest by the first stain, the secondfeatures of interest being different from the first features ofinterest. In some instances, the second stain may be one of the same asthe first stain but used to stain the second features of interest ordifferent from the first stain.

According to some embodiments, the first image may comprise one of afirst set of color or brightfield images, a first fluorescence image, afirst phase image, or a first spectral image, and/or the like, whereinthe first set of color or brightfield images may comprise a firstred-filtered (“R”) image, a first green-filtered (“G”) image, and afirst blue-filtered (“B”) image, and/or the like. The first fluorescenceimage may comprise at least one of a first autofluorescence image or afirst labelled fluorescence image having one or more channels withdifferent excitation or emission characteristics, and/or the like. Thefirst spectral image may comprise at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, and/or the like. Similarly, the second image maycomprise one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like, wherein the second set of color or brightfield imagesmay comprise a second R image, a second G image, and a second B image,and/or the like. The second fluorescence image may comprise at least oneof a second autofluorescence image or a second labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may compriseat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, and/or the like.

In some embodiments, the first set of instructions, when executed by theat least one first processor, may further cause the computing system to:receive a third image of the first biological sample, the third imagecomprising a third FOV of the first biological sample that has beenstained with a third stain different from each of the first stain andthe second stain; and autonomously perform one of: aligning the thirdimage with each of the first image and the second image images to createsecond aligned images, by aligning one or more features of interest inthe first biological sample as depicted in the third image with the sameone or more features of interest in the first biological sample asdepicted in each of the first image and the second image; or aligningthe third image with the first aligned images to create second alignedimages, by aligning one or more features of interest in the firstbiological sample as depicted in the third image with the same one ormore features of interest in the first biological sample as depicted inthe first aligned images. The first set of instructions, when executedby the at least one first processor, may further cause the computingsystem to: autonomously create second aligned image patches from thesecond aligned images, by extracting a portion of the second alignedimages, the portion of the second aligned images comprising a firstpatch corresponding to the extracted portion of the first image, asecond patch corresponding to the extracted portion of the second image,and a third patch corresponding to the extracted portion of the thirdimage; train the AI system to update Model G* to generate secondinstance classification of features of interest in the first biologicalsample, based at least in part on the second aligned image patches andthe labeling of instances of features of interest contained in the firstset of image patches; and train the AI system to update Model G toidentify instances of features of interest in the first biologicalsample, based at least in part on the first patch and the secondinstance classification of features of interest generated by Model G*.

According to some embodiments, the first set of instructions, whenexecuted by the at least one first processor, may further cause thecomputing system to: receive a fourth image, the fourth image comprisingone of a fourth FOV of the first biological sample different from thefirst FOV and the second FOV or a fifth FOV of a second biologicalsample, wherein the second biological sample is different from the firstbiological sample; and identify, using Model G, second instances offeatures of interest in the second biological sample, based at least inpart on the fourth image and based at least in part on training of ModelG using the first patch and the first instance classification offeatures of interest generated by Model G*. According to someembodiments, the first set of instructions, when executed by the atleast one first processor, may further cause the computing system to:generate, using Model G*, a clinical score, based at least in part onthe identified second instances of features of interest.

In yet another aspect, a method may comprise receiving, with a computingsystem, a first image of a first biological sample, the first imagecomprising a first field of view (“FOV”) of the first biological sample;and identifying, using a first artificial intelligence (“AI”) model(“Model G”) that is generated or updated by a trained AI system, firstinstances of features of interest in the first biological sample, basedat least in part on the first image and based at least in part ontraining of Model G using a first patch and first instanceclassification of features of interest generated by a second model(Model G*) that is generated or updated by the trained AI system byusing first aligned image patches and labeling of instances of featuresof interest contained in the first patch. In some cases, the firstaligned image patches may comprise an extracted portion of first alignedimages. The first aligned images may comprise a second image and a thirdimage that have been aligned. The second image may comprise a second FOVof a second biological sample that has been stained with a first stain,the second biological sample being different from the first biologicalsample. The third image may comprise a third FOV of the secondbiological sample that has been stained with a second stain. The secondpatch may comprise labeling of instances of features of interest asshown in the extracted portion of the second image of the first alignedimages. The first image may comprise highlighting of first features ofinterest in the second biological sample by the first stain that hadbeen applied to the second biological sample. The second patch maycomprise one of highlighting of second features of interest in thesecond biological sample by the second stain that had been applied tothe second biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain.

In still another aspect, a system may comprise a computing system, whichmay comprise at least one first processor and a first non-transitorycomputer readable medium communicatively coupled to the at least onefirst processor. The first non-transitory computer readable medium mayhave stored thereon computer software comprising a first set ofinstructions that, when executed by the at least one first processor,causes the computing system to: receive a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample; and identify, using a firstartificial intelligence (“AI”) model (“Model G”) that is generated orupdated by a trained AI system, first instances of features of interestin the first biological sample, based at least in part on the firstimage and based at least in part on training of Model G using a firstpatch and first instance classification of features of interestgenerated by a second model (Model G*) that is generated or updated bythe trained AI system by using first aligned image patches and labelingof instances of features of interest contained in the first patch,wherein the first aligned image patches may comprise an extractedportion of first aligned images, wherein the first aligned images maycomprise a second image and a third image that have been aligned,wherein the second image may comprise a second FOV of a secondbiological sample that has been stained with a first stain, the secondbiological sample being different from the first biological sample,wherein the third image may comprise a third FOV of the secondbiological sample that has been stained with a second stain, wherein thesecond patch may comprise labeling of instances of features of interestas shown in the extracted portion of the second image of the firstaligned images, wherein the first image may comprise highlighting offirst features of interest in the second biological sample by the firststain that had been applied to the second biological sample wherein thesecond patch may comprise one of highlighting of second features ofinterest in the second biological sample by the second stain that hadbeen applied to the second biological sample in addition to highlightingof the first features of interest by the first stain or highlighting ofthe second features of interest by the second stain without highlightingof the first features of interest by the first stain.

In yet another aspect, a method may comprise receiving, with a computingsystem, first instance classification of features of interest (“GroundTruth”) in a first biological sample that has been sequentially stained,the Ground Truth having been generated by a trained first model (“ModelG*”) that has been trained or updated by an artificial intelligence(“AI”) system, wherein the Ground Truth is generated by using firstaligned image patches and labeling of instances of features of interestcontained in the first aligned image patches, wherein the first alignedimage patches comprise an extracted portion of first aligned images,wherein the first aligned images comprise a first image and a secondimage that have been aligned, wherein the first image comprises a firstFOV of the first biological sample that has been stained with a firststain, wherein the second image comprises a second FOV of the firstbiological sample that has been stained with a second stain, wherein thefirst aligned image patches comprise labeling of instances of featuresof interest as shown in the extracted portion of the first alignedimages; and training, using the AI system, a second AI model (“Model G”)to identify instances of features of interest in the first biologicalsample, based at least in part on the first instance classification offeatures of interest generated by Model G*.

In still another aspect, a system may comprise a computing system, whichmay comprise at least one first processor and a first non-transitorycomputer readable medium communicatively coupled to the at least onefirst processor. The first non-transitory computer readable medium mayhave stored thereon computer software comprising a first set ofinstructions that, when executed by the at least one first processor,causes the computing system to: receive first instance classification offeatures of interest (“Ground Truth”) in a first biological sample thathas been sequentially stained, the Ground Truth having been generated bya trained first model (“Model G*”) that has been trained or updated byan artificial intelligence (“AI”) system, wherein the Ground Truth isgenerated by using first aligned image patches and labeling of instancesof features of interest contained in the first aligned image patches,wherein the first aligned image patches comprise an extracted portion offirst aligned images, wherein the first aligned images comprise a firstimage and a second image that have been aligned, wherein the first imagecomprises a first FOV of the first biological sample that has beenstained with a first stain, wherein the second image comprises a secondFOV of the first biological sample that has been stained with a secondstain, wherein the first aligned image patches comprise labeling ofinstances of features of interest as shown in the extracted portion ofthe first aligned images; and train, using the AI system, a secondartificial intelligence (“AI”) model (“Model G”) to identify instancesof features of interest in the first biological sample, based at leastin part on the first. instance classification of features of interestgenerated by Model G*.

In yet another aspect, a method may comprise generating, using acomputing system, ground truth for developing accurate artificialintelligence (“AI”) models for biological image interpretation, based atleast in part on images of a first biological sample depictingsequential staining of the first biological sample.

In an aspect, a method may comprise receiving, with a computing system,a first image of a first biological sample, the first image comprising afirst field of view (“FOV”) of the first biological sample that has beenstained with a first stain; receiving, with the computing system, asecond image of the first biological sample, the second image comprisinga second FOV of the first biological sample that has been stained withat least a second stain; and aligning, with the computing system, thefirst image with the second image to create first aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the first image with the same one or more features ofinterest in the first biological sample as depicted in the second image.

The method may further comprise autonomously creating, with thecomputing system, first aligned image patches from the first alignedimages, by extracting a portion of the first aligned images, the portionof the first aligned images comprising a first patch corresponding tothe extracted portion of the first image and a second patchcorresponding to the extracted portion of the second image; training,using an artificial intelligence (“AI”) system, a first AI model (“ModelF”) to generate a third patch comprising a virtual stain of the firstaligned image patches, based at least in part on the first patch and thesecond patch, the virtual stain simulating staining by at least thesecond stain of features of interest in the first biological sample asshown in the second patch; and training, using the AI system, a secondmodel (“Model G*”) to identify or classify first instances of featuresof interest in the first biological sample, based at least in part onthe third patch and based at least in part on results from an externalinstance classification process or a region of interest detectionprocess.

According to some embodiments, the computing system may comprise one ofa computing system disposed in a work environment, a remote computingsystem disposed external to the work environment and accessible over anetwork, a web server, a web browser, or a cloud computing system,and/or the like. In some cases, the work environment may comprise atleast one of a laboratory, a clinic, a medical facility, a researchfacility, a healthcare facility, or a room, and/or the like. In someinstances, the AI system may comprise at least one of a machine learningsystem, a deep learning system, a model architecture, a statisticalmodel-based system, or a deterministic analysis system, and/or the like.In some instances, the model architecture may comprise at least one of aneural network, a convolutional neural network (“CNN”), or a fullyconvolutional network (“FCN”), and/or the like.

In some embodiments, the first biological sample may comprise one of ahuman tissue sample, an animal tissue sample, a plant tissue sample, oran artificially produced tissue sample, and/or the like. In someinstances, the features of interest may comprise at least one of normalcells, abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, or organ structures, and/or the like.

According to some embodiments, the first image may comprise highlightingof first features of interest in the first biological sample by thefirst stain that had been applied to the first biological sample. Insome cases, the third patch may comprise one of highlighting of thefirst features of interest by the first stain and highlighting of secondfeatures of interest in the first biological sample by the virtual stainthat simulates the second stain having been applied to the firstbiological sample or highlighting of the first features of interest bythe first stain and highlighting of first features of interest by thevirtual stain, the second features of interest being different from thefirst features of interest. In some instances, the second stain may beone of the same as the first stain but used to stain the second featuresof interest or different from the first stain.

Merely by way of example, in some cases, the first stain may comprise atleast one of Hematoxylin, Acridine orange, Bismarck brown, Carmine,Coomassie blue, Cresyl violet, Crystal violet,4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.Similarly, the second stain may comprise at least one of Hematoxylin,Acridine orange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet,Crystal violet, DAPI, Eosin, Ethidium bromide intercalates, Acidfuchsine, Hoechst stain, Iodine, Malachite green, Methyl green,Methylene blue, Neutral red, Nile blue, Nile red, Osmium tetroxide,Propidium Iodide, Rhodamine, Safranine, PD-L1 stain, DAB chromogen,Magenta chromogen, cyanine chromogen, CD3 stain, CD20 stain, CD68 stain,p40 stain, antibody-based stain, or label-free imaging marker, and/orthe like.

In some embodiments, the first image may comprise one of a first set ofcolor or brightfield images, a first fluorescence image, a first phaseimage, or a first spectral image, and/or the like, wherein the first setof color or brightfield images may comprise a first red-filtered (“R”)image, a first green-filtered (“G”) image, and a first blue-filtered(“B”) image, and/or the like. The first fluorescence image may compriseat least one of a first autofluorescence image or a first labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The first spectral imagemay comprise at least one of a first Raman spectroscopy image, a firstnear infrared (“NIR”) spectroscopy image, a first multispectral image, afirst hyperspectral image, or a first full spectral image, and/or thelike. In some instances, the second image may comprise one of a secondset of color or brightfield images, a second fluorescence image, asecond phase image, or a second spectral image, and/or the like, whereinthe second set of color or brightfield images may comprise a second Rimage, a second G image, and a second B image, and/or the like. Thesecond fluorescence image may comprise at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may compriseat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, and/or the like. In some cases,the third patch may comprise one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, wherein the third set of color orbrightfield images may comprise a third R image, a third G image, and athird B image, and/or the like. The third fluorescence image maycomprise at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may comprise at least one of a third Raman spectroscopyimage, a third NIR spectroscopy image, a third multispectral image, athird hyperspectral image, or a third full spectral image, and/or thelike.

According to some embodiments, aligning the first image with the secondimage may be one of performed manually using manual inputs to thecomputing system or performed autonomously, wherein autonomous alignmentmay comprise alignment using at least one of automated global alignmenttechniques or optical flow alignment techniques.

In some embodiments, training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample may comprise training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample, based at least in part on one or more of the first patch, thesecond patch, or the third patch and based at least in part on theresults from the external instance classification process or the regionof interest detection process. In some cases, the external instanceclassification process or the region of interest detection process mayeach comprise at least one of detection of nuclei in the first image orthe first patch by a nuclei detection method, identification of nucleiin the first image or the first patch by a user (e.g., a pathologist, orother domain expert, or the like), detection of features of interest inthe first image or the first patch by a feature detection method, oridentification of features of interest in the first image or the firstpatch by the pathologist, and/or the like.

According to some embodiments, training the AI system to update Model Fmay comprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may compriseone of a mean squared error loss function, a mean squared logarithmicerror loss function, a mean absolute error loss function, a Huber lossfunction, or a weighted sum of squared differences loss function, and/orthe like.

In some embodiments, the first patch may comprise one of a first set ofcolor or brightfield images, a first fluorescence image, a first phaseimage, or a first spectral image, and/or the like, wherein the first setof color or brightfield images may comprise a first red-filtered (“R”)image, a first green-filtered (“G”) image, and a first blue-filtered(“B”) image, and/or the like. The first fluorescence image may compriseat least one of a first autofluorescence image or a first labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The first spectral imagemay comprise at least one of a first Raman spectroscopy image, a firstnear infrared (“NIR”) spectroscopy image, a first multispectral image, afirst hyperspectral image, or a first full spectral image, and/or thelike. In some cases, the second image may comprise one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, and/or the like, wherein thesecond set of color or brightfield images may comprise a second R image,a second G image, and a second B image, and/or the like. The secondfluorescence image may comprise at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may compriseat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, and/or the like. In someinstances, the predicted virtually stained image patch may comprise oneof a third set of color or brightfield images, a third fluorescenceimage, a third phase image, or a third spectral image, and/or the like,wherein the third set of color or brightfield images may comprise athird R image, a third G image, and a third B image, and/or the like.The third fluorescence image may comprise at least one of thirdautofluorescence image or a third labelled fluorescence image having oneor more channels with different excitation or emission characteristics,and/or the like. The third spectral image may comprise at least one of athird Raman spectroscopy image, a third NIR spectroscopy image, a thirdmultispectral image, a third hyperspectral image, or a third fullspectral image, and/or the like.

According to some embodiments, operating on the encoded first patch togenerate the color vector may comprise operating on the encoded firstpatch to generate a color vector, using a color vector output and one ofa model architecture, a statistical model-based system, or adeterministic analysis system, and/or the like, wherein the modelarchitecture may comprise at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”), and/or the like, wherein the color vector may be one of a fixedcolor vector or a learned color vector that is not based on the encodedfirst patch, and/or the like.

In some embodiments, the image of the virtual stain may comprise one ofa 3-channel image of the virtual stain, a RGB-transform image of thevirtual stain, or a logarithmic transform image of the virtual stain,and/or the like. In some instances, generating the predicted virtuallystained image patch may comprise adding the generated image of thevirtual stain to the received first patch to produce the predictedvirtually stained image patch.

According to some embodiments, the first loss value may comprise one ofa pixel loss value between each pixel in the predicted virtually stainedimage patch and a corresponding pixel in the second patch or agenerative adversarial network (“GAN”) loss value between the predictedvirtually stained image patch and the second patch, and/or the like,wherein the GAN loss value may be generated based on one of a minimaxGAN loss function, a non-saturating GAN loss function, a least squaresGAN loss function, or a Wasserstein GAN loss function, and/or the like.

Alternatively, training the AI system to update Model F may comprise:receiving, with the AI system, the first patch; receiving, with the AIsystem, the second patch; generating, with a second model of the AIsystem, an image of the virtual stain; adding the generated image of thevirtual stain to the received first patch to produce a predictedvirtually stained image patch; determining a first loss value betweenthe predicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value.

In some embodiments, the second model may comprise at least one of aconvolutional neural network (“CNN”), a U-Net, an artificial neuralnetwork (“ANN”), a residual neural network (“ResNet”), an encode/decodeCNN, an encode/decode U-Net, an encode/decode ANN, or an encode/decodeResNet, and/or the like.

According to some embodiments, the method may further comprisereceiving, with the computing system, a fourth image, the fourth imagecomprising one of a fourth FOV of the first biological sample differentfrom the first FOV and the second FOV or a fifth FOV of a secondbiological sample, wherein the second biological sample is differentfrom the first biological sample; and identifying, using Model G*,second instances of features of interest in the second biologicalsample, based at least in part on the fourth image and based at least inpart on training of Model G* using at least the third patch comprisingthe virtual stain of the first aligned image patches. According to someembodiments, the method may further comprise generating, using Model G,a clinical score, based at least in part on the identified secondinstances of features of interest.

In another aspect, a system may comprise a computing system, which maycomprise at least one first processor and a first non-transitorycomputer readable medium communicatively coupled to the at least onefirst processor. The first non-transitory computer readable medium mayhave stored thereon computer software comprising a first set ofinstructions that, when executed by the at least one first processor,causes the computing system to: receive a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample that has been stained with afirst stain; receive a second image of the first biological sample, thesecond image comprising a second FOV of the first biological sample thathas been stained with at least a second stain; align the first imagewith the second image to create first aligned images, by aligning one ormore features of interest in the first biological sample as depicted inthe first image with the same one or more features of interest in thefirst biological sample as depicted in the second image; autonomouslycreate first aligned image patches from the first aligned images, byextracting a portion of the first aligned images, the portion of thefirst aligned images comprising a first patch corresponding to theextracted portion of the first image and a second patch corresponding tothe extracted portion of the second image; train, using an artificialintelligence (“AI”) system, a first AI model (“Model F”) to generate athird patch comprising a virtual stain of the first aligned imagepatches, based at least in part on the first patch and the second patch,the virtual stain simulating staining by at least the second stain offeatures of interest in the first biological sample as shown in thesecond patch; and train, using the AI system, a second model (“ModelG*”) to identify or classify first instances of features of interest inthe first biological sample, based at least in part on the third patchand based at least in part on results from an external instanceclassification process or a region of interest detection process.

In some embodiments, the computing system may comprise one of acomputing system disposed in a work environment, a remote computingsystem disposed external to the work environment and accessible over anetwork, a web server, a web browser, or a cloud computing system,and/or the like. In some cases, the work environment may comprise atleast one of a laboratory, a clinic, a medical facility, a researchfacility, a healthcare facility, or a room, and/or the like. In someinstances, the AI system may comprise at least one of a machine learningsystem, a deep learning system, a model architecture, a statisticalmodel-based system, or a deterministic analysis system, and/or the like.In some instances, the model architecture may comprise at least one of aneural network, a convolutional neural network (“CNN”), or a fullyconvolutional network (“FCN”), and/or the like.

According to some embodiments, the first biological sample may compriseone of a human tissue sample, an animal tissue sample, a plant tissuesample, or an artificially produced tissue sample, and/or the like,wherein the features of interest may comprise at least one of normalcells, abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, or organ structures, and/or the like.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample by the first stainthat had been applied to the first biological sample. In some cases, thethird patch may comprise one of highlighting of the first features ofinterest by the first stain and highlighting of second features ofinterest in the first biological sample by the virtual stain thatsimulates the second stain having been applied to the first biologicalsample or highlighting of the first features of interest by the firststain and highlighting of first features of interest by the virtualstain, the second features of interest being different from the firstfeatures of interest. In some instances, the second stain may be one ofthe same as the first stain but used to stain the second features ofinterest or different from the first stain.

According to some embodiments, the first image may comprise one of afirst set of color or brightfield images, a first fluorescence image, afirst phase image, or a first spectral image, and/or the like, whereinthe first set of color or brightfield images may comprise a firstred-filtered (“R”) image, a first green-filtered (“G”) image, and afirst blue-filtered (“B”) image, and/or the like. The first fluorescenceimage may comprise at least one of a first autofluorescence image or afirst labelled fluorescence image having one or more channels withdifferent excitation or emission characteristics, and/or the like. Thefirst spectral image may comprise at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, and/or the like. In some instances, the second image maycomprise one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like, wherein the second set of color or brightfield imagesmay comprise a second R image, a second G image, and a second B image,and/or the like. The second fluorescence image may comprise at least oneof a second autofluorescence image or a second labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may compriseat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, and/or the like. In some cases,the third patch may comprise one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, wherein the third set of color orbrightfield images may comprise a third R image, a third G image, and athird B image, and/or the like. The third fluorescence image maycomprise at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may comprise at least one of a third Raman spectroscopyimage, a third NIR spectroscopy image, a third multispectral image, athird hyperspectral image, or a third full spectral image, and/or thelike.

In some embodiments, training the AI system to update Model F maycomprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may compriseone of a mean squared error loss function, a mean squared logarithmicerror loss function, a mean absolute error loss function, a Huber lossfunction, or a weighted sum of squared differences loss function, and/orthe like.

According to some embodiments, the first loss value may comprise one ofa pixel loss value between each pixel in the predicted virtually stainedimage patch and a corresponding pixel in the second patch or agenerative adversarial network (“GAN”) loss value between the predictedvirtually stained image patch and the second patch, and/or the like,wherein the GAN loss value may be generated based on one of a minimaxGAN loss function, a non-saturating GAN loss function, a least squaresGAN loss function, or a Wasserstein GAN loss function, and/or the like.

In some embodiments, the first set of instructions, when executed by theat least one first processor, may further cause the computing system to:receive a fourth image, the fourth image comprising one of a fourth FOVof the first biological sample different from the first FOV and thesecond FOV or a fifth FOV of a second biological sample, wherein thesecond biological sample is different from the first biological sample;and identify, using Model G*, second instances of features of interestin the second biological sample, based at least in part on the fourthimage and based at least in part on training of Model G* using at leastthe third patch comprising the virtual stain of the first aligned imagepatches. According to some embodiments, the first set of instructions,when executed by the at least one first processor, may further cause thecomputing system to: generate, using Model G*, a clinical score, basedat least in part on the identified second instances of features ofinterest.

In yet another aspect, a method may comprise receiving, with a computingsystem, a first image of a first biological sample, the first imagecomprising a first field of view (“FOV”) of the first biological sample;and identifying, using a first model (“Model G*”) that is generated orupdated by a trained AI system, first instances of features of interestin the first biological sample, based at least in part on the firstimage and based at least in part on training of Model G* using at leasta first patch comprising a virtual stain of first aligned image patches,the first patch being generated by a second artificial intelligence(“AI”) model (Model F) that is generated or updated by the trained AIsystem by using a second patch. In some cases, the first aligned imagepatches may comprise an extracted portion of first aligned images. Thefirst aligned images may comprise a second image and a third image thathave been aligned. The second image may comprise a second FOV of asecond biological sample that is different from the first biologicalsample that has been stained with a first stain. The second patch maycomprise the extracted portion of the second image. The third image maycomprise a third FOV of the second biological sample that has beenstained with at least a second stain.

In still another aspect, a system may comprise a computing system, whichmay comprise at least one first processor and a first non-transitorycomputer readable medium communicatively coupled to the at least onefirst processor. The first non-transitory computer readable medium mayhave stored thereon computer software comprising a first set ofinstructions that, when executed by the at least one first processor,causes the computing system to: receive a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample; and identify, using a firstmodel (“Model G*”) that is generated or updated by a trained AI system,first instances of features of interest in the first biological sample,based at least in part on the first image and based at least in part ontraining of Model G* using at least a first patch comprising a virtualstain of first aligned image patches, the first patch being generated bya second artificial intelligence (“AI”) model (Model F) that isgenerated or updated by the trained AI system by using a second patch,wherein the first aligned image patches may comprise an extractedportion of first aligned images, wherein the first aligned images maycomprise a second image and a third image that have been aligned,wherein the second image may comprise a second FOV of a secondbiological sample that is different from the first biological samplethat has been stained with a first stain, wherein the second patch maycomprise the extracted portion of the second image, wherein the thirdimage may comprise a third FOV of the second biological sample that hasbeen stained with at least a second stain.

Various modifications and additions can be made to the embodimentsdiscussed without departing from the scope of the invention. Forexample, while the embodiments described above refer to particularfeatures, the scope of this invention also includes embodiments havingdifferent combination of features and embodiments that do not includeall of the above described features.

SPECIFIC EXEMPLARY EMBODIMENTS I. Tissue Staining Methodology

In the following detailed description, for purposes of explanation andnot limitation, representative embodiments disclosing specific detailsare set forth in order to provide a thorough understanding of thepresent teachings. Descriptions or details of known systems, compounds,materials, methods of use and methods of manufacture may be omitted soas to avoid obscuring the description of the example embodiments.Nonetheless, systems, moieties and methods that are within the purviewof one of ordinary skill in the art may be used in accordance with therepresentative embodiments.

In many methods for diagnosing a disease or disorder or for determiningan appropriate therapy for an individual suffering from a disease ordisorder, it is desirable to detect multiple target molecules in abiological sample. In some embodiments, the target molecules may bemolecules which are indicative of a disease state such as a tumor ormolecules indicative of particular cell types, such as immune cells. Forexample, in some methods, such as those described in U.S. Pat. No.10,613,092 (the disclosure of which is incorporated herein in itsentirety), the number of PD-L1 positive tumor cells and the number ofPD-L1 mononuclear inflammatory cells in a tumor tissue may beidentified. Such methods may utilize antibodies which specifically bindto PD-L1 as well as antibodies which recognize target molecules found onmononuclear inflammatory cells, such as CD3,CD5,CD4, CD7, CD8, CD20 orCD68 in order to identify PD-L1 positive tumor cells and PD-L1 positivemononuclear inflammatory cells. Based on the results of this analysis,an appropriate therapy may be recommended for a particular individual.In some embodiments, the methods and compounds described herein may beused to sequentially detect cells expressing PD-L1 and mononuclearinflammatory cells such as lymphocytes or to develop digital imaginganalyses which can be used to perform diagnostic methods or to assist inrecommending an appropriate therapy.

In one aspect, a method for generating cell-level annotations from abiological sample comprising cells comprises: exposing the biologicalsample to a first ligand that recognizes a first antigen thereby forminga first ligand-antigen complex; exposing the first ligand-antigencomplex to a first labeling reagent binding to the first ligand, thefirst labeling reagent forming a first detectable reagent, whereby thefirst detectable reagent is precipitated around the first antigen andvisible in brightfield; exposing the biological sample to a secondligand that recognizes a second antigen thereby forming a secondligand-antigen complex; exposing the second ligand-antigen complex to asecond labeling reagent binding to the second ligand, the secondlabeling reagent comprising a substrate not visible in brightfield,whereby the substrate is precipitated around the second antigen;obtaining a first image of the biological sample in brightfield tovisualize the first detectable reagent precipitated in the biologicalsample using a detection method; exposing the biological sample to athird labeling reagent that recognizes the substrate, thereby forming athird ligand-antigen complex, the third labeling reagent forming asecond detectable reagent, whereby the second detectable reagent isprecipitated around the second antigen; obtaining a second image of thebiological sample at a wavelength with the second detectable reagentprecipitated in the biological sample using the detection method;creating a mask from the second image; applying the mask to the firstimage so as to obtain an image of the biological sample annotated withthe second antigen.

In one aspect, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample, wherein theobjects of interest comprise at least one of normal cells, cell type,abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, or organ structures.

In one aspect, the detectable reagent is a chromogen. In one aspect thevisibility or invisibility in brightfield can be in parts of or thewhole brightfield spectra. In one aspect, the obtaining of the image canbe through using a digital scanner.

In one aspect, the method further comprises, applying a target retrievalbuffer and protein blocking solution to the biological sample, wherebythe first and second antigens are retrieved for the subsequent steps. Inone aspect the applying a target retrieval buffer and protein blockingsolution to the biological sample occurs prior to exposure of thebiological sample.

In one aspect, the method comprises the first and third labelingreagents comprising an enzyme that acts on a detectable reagent to formthe first and second detectable reagents, respectively. In one aspect,the detectable reagents are chromogenic substrates to form the first andsecond chromogens, respectively.

In one aspect, the method further comprises the first and/or secondantigens are selected from the group consisting of non-nuclear proteins,non-nuclear antigens, membranal antigens, cytoplasmic antigens, andnuclear antigens.

In one aspect, method comprises, following exposing the firstligand-antigen complex to a first labeling reagent binding to the firstligand, reversing the binding of the first ligand. In one aspect, thereversing the binding can be through denaturing.

In one aspect, the method further comprises, following exposing thesecond detectable reagent to a second labeling reagent binding to thesecond ligand, i) counterstaining cell nuclei of the biological sample,and ii) dehydrating and mounting the sample on a slide using a mountingmedium.

In one aspect, the method further comprises, following obtaining a firstimage of the biological sample in brightfield to visualize the firstchromogen precipitated in the biological sample, i) removing mountingmedium from the slide, and ii) rehydrating the biological sample.

In one aspect, the method further comprises, following exposing thebiological sample to a third labeling reagent that recognizes thesubstrate, i) counterstaining cell nuclei of the biological sample, andii) dehydrating and mounting the sample on a slide.

In one aspect, the method further comprising the first antigen comprisesa lymphocyte-specific antigen, the first ligand comprises an anti-lymphocyte-specific antigen antibody (“primary antibody”), the firstlabeling reagent comprises a horseradish peroxidase (HRP)-coupledpolymer capable of binding to the primary antibody, the first detectablereagent comprises an HRP substrate comprising HRP Magenta (availablefrom Agilent Technologies, Inc.) or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises PD-L1, the secondligand comprises anti-PD-L1 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-PD-L1antibodies, the substrate comprises fluorescein-coupled longsingle-chained polymer (fer-4-flu linker), third labeling reagent ananti-FITC antibody coupled to HRP, and the second detectable reagentcomprises HRP Magenta or DAB. It will be appreciated that the antigensmay be stained in any order. For example, the lymphocyte-specificantigen may be stained first, followed by PD-L1. Alternatively, PD-L1may be stained first, followed by the lymphocyte-specific antigen.

In one aspect, the first antigen comprises a lymphocyte-specificantigen, the first labeling reagent comprises a horseradish peroxidase(HRP)-coupled polymer capable of binding to the primary antibody, thefirst detectable reagent comprises an HRP substrate comprising HRPMagenta or 3,3′-diaminobenzidine tetrahydrochloride (DAB), the secondantigen comprises p40, the second ligand comprises anti-p40 antibodies,the second labeling reagent comprises an HRP-coupled polymer capable ofbinding to anti-p40 antibodies, the substrate comprisesfluorescein-coupled long single-chained polymer (fer-4-flu linker), thethird labeling reagent comprises an anti-FITC antibody coupled to HRP,and the second detectable reagent comprises HRP Magenta or DAB. In oneaspect, an annotated image is obtained. It will be appreciated that theantigens may be stained in any order. For example, the p40 antigen maybe stained first, followed by the lymphocyte-specific antigen.Alternatively, the lymphocyte-specific antigen may be stained first,followed by the p40 antigen.

In some embodiments, the detectable reagents of the present disclosureabsorb and/or emit light in the range from about 450 nm to about 600 nm.In some embodiments, the detectable reagent absorbs light at 536 nm andthe color of stain may be defined as magenta. In some embodiments, thedetectable reagent is yellow and absorbs light at 450 nm. In someembodiments, the detectable reagent is purple and absorbs light close to600 nm. In some embodiments, the present detectable reagents can serveas substrates of a peroxidase enzyme, e.g. HRP, and they are spectrallynarrow, non-dichromatic, and do not change their spectralcharacteristics upon precipitation. In some embodiments, the stainsproduced via enzymatic precipitation of the detectable reagents arepoorly soluble, if at all, in water or organic solutions and do notbleach when exposed to light sources used for imaging of stainedsamples. These features make the present detectable reagentsparticularly suitable for automated image analyses and multiplexing.Further, the molecules of the detectable reagents have well-definedchemical structures.

In one aspect, a method for generating cell-level annotations from abiological sample comprising cells comprises: exposing the biologicalsample to a first ligand that recognizes a first antigen thereby forminga first ligand-antigen complex; exposing the first ligand-antigencomplex to a first labeling reagent binding to the first ligand, thefirst labeling reagent forming a first detectable reagent, whereby thefirst detectable reagent is precipitated around the first antigen;exposing the biological sample to a second ligand that recognizes asecond antigen thereby forming a second ligand-antigen complex; exposingthe second ligand-antigen complex to a second labeling reagent bindingto the second ligand, the second labeling reagent forming a seconddetectable reagent, whereby the second detectable reagent isprecipitated around the second antigen; obtaining a first image of thebiological sample with the first and second detectable reagentsprecipitated in the biological sample an image is obtained; incubatingthe tissue sample with an agent which dissolves the second detectablereagent; obtaining a second image of the biological sample with thefirst detectable reagent precipitated in the biological sample; creatinga mask from the first image; and applying the mask to the second imageso as to obtain an annotated image of the biological sample with thesecond antigen.

In one aspect, the method further comprises, wherein the firstbiological sample comprises one of a human tissue sample, an animaltissue sample, or a plant tissue sample, wherein the objects of interestcomprise at least one of normal cells, abnormal cells, damaged cells,cancer cells, tumors, subcellular structures, or organ structures.

In one aspect, the method further comprises, applying a target retrievalbuffer and protein blocking solution to the biological sample, wherebythe first and second antigens are exposed for the subsequent steps andendogenous peroxidases are inactivated. In one aspect, the first andsecond labeling reagents comprise an enzyme that acts on a detectablesubstrate to form the first and second detectable reagents,respectively.

In one aspect, the method comprises denaturing the first ligands toretrieve the first antigens available. In one aspect, the method furthercomprises, following exposing the second ligand-antigen complex to asecond labeling reagent binding to the second ligand, counterstainingcell nuclei of the biological sample, and dehydrating and mounting thesample on a slide using a mounting medium. In one aspect, the methodfurther comprises, following obtaining a first image of the biologicalsample, removing mounting medium from the slide, and rehydrating thebiological sample. In one aspect, the method further comprises,dehydrating and mounting the sample on a slide. In one aspect, the firstantigen comprises a lymphocyte-specific antigen, the first ligandcomprises an anti-lymphocyte-specific antigen antibody (“primaryantibody”), the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises PD-L1 the second ligand comprisesanti-PD-L1 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, thesecond detectable reagent comprises amino ethyl carbazole (AEC), and theagent which dissolves the second detectable reagent is alcohol oracetone. It will be appreciated that the antigens may be stained in anyorder. For example, the lymphocyte-specific antigen may be stainedfirst, followed by PD-L1. Alternatively, PD-L1 may be stained first,followed by the lymphocyte-specific antigen.

In one aspect, the first antigen comprises a lymphocyte-specificantigen, the first ligand comprises an anti-lymphocyte-specific antigenantibody (“primary antibody”), the first labeling reagent comprises ahorseradish peroxidase (HRP)-coupled polymer capable of binding to theprimary antibody, the first detectable reagent comprises an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises p40 the secondligand comprises anti-p40 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-p40antibodies, the second detectable reagent comprises amino ethylcarbazole (AEC), and the agent which dissolves the second detectablereagent is alcohol or acetone. It will be appreciated that the antigensmay be stained in any order. For example, the p40 antigen may be stainedfirst, followed by the lymphocyte-specific antigen. Alternatively, thelymphocyte-specific antigen may be stained first, followed by the p40antigen. In one aspect, the counterstaining agent is hematoxylin. In oneaspect, an annotated image is obtained.

In one aspect, a method for generating cell-level annotations from abiological sample comprising cells comprises: exposing the biologicalsample to a first ligand that recognizes a first antigen thereby forminga first ligand-antigen complex; exposing the first ligand-antigencomplex to a first labeling reagent binding to the first ligand, thefirst labeling reagent forming a first detectable reagent, wherein thefirst detectable reagent is visible in brightfield; whereby the firstdetectable reagent is precipitated around the first antigen; exposingthe biological sample to a second ligand that recognizes a secondantigen thereby forming a second ligand-antigen complex; exposing thesecond ligand-antigen complex to a second labeling reagent binding tothe second ligand, the second labeling reagent forming a seconddetectable reagent, wherein the second detectable reagent is visible influorescence; whereby the second detectable reagent is precipitatedaround the second antigen; obtaining a first brightfield image of thebiological sample with the first detectable reagent precipitated in thebiological sample; obtaining a second fluorescent image of thebiological sample with the second detectable reagent precipitated in thebiological sample; creating a mask from the second image; applying themask to the first image so as to obtain an annotated image of the tissuesample with the second marker.

In one aspect, the first biological sample comprises one of a humantissue sample, an animal tissue sample, or a plant tissue sample,wherein the objects of interest comprise at least one of normal cells,abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, or organ structures. In one aspect, applying a targetretrieval buffer and protein blocking solution to the biological sample,whereby the first and second antigens are exposed for the subsequentsteps and endogenous peroxidases are inactivated.

In one aspect, the first and second labeling reagents comprise an enzymethat acts on a detectable substrate to form the first and seconddetectable reagents, respectively. In one aspect, the antigens areselected from the group consisting of non-nuclear proteins, non-nuclearantigens, membranal antigens, cytoplasmic antigens, and nuclearantigens. In one aspect, further comprising denaturing the first ligandsto retrieve the first antigens available. In one aspect, furthercomprising, following exposing the first ligand-antigen complex,counterstaining cell nuclei of the biological sample, and dehydratingand mounting the sample on a slide using a mounting medium.

In one aspect, the first antigen comprises a lymphocyte-specificantigen, the first ligand comprises an anti-lymphocyte-specific antigenantibody (“primary antibody”), the first labeling reagent comprises ahorseradish peroxidase (HRP)-coupled polymer capable of binding to theprimary antibody, the first detectable reagent comprises an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises PD-L, the secondligand comprises anti-PD-L1 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-PD-L1antibodies, and the second detectable reagent comprises arhodamine-based fluorescent compound coupled to a long single-chainpolymer. It will be appreciated that the antigens may be stained in anyorder. For example, the lymphocyte-specific antigen may be stainedfirst, followed by PD-L1. Alternatively, PD-L1 may be stained first,followed by the lymphocyte-specific antigen. In one aspect, firstantigen comprises a lymphocyte-specific antigen, the first ligandcomprises an anti-lymphocyte-specific antigen antibody (“primaryantibody”), the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises p40,the second ligand comprisesanti-p40 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-p40 antibodies, and thesecond detectable reagent comprises rhodamine-based fluorescent compoundcoupled to a long single-chain polymer. It will be appreciated that theantigens may be stained in any order. For example, the p40 antigen maybe stained first, followed by the lymphocyte-specific antigen.Alternatively, the lymphocyte-specific antigen may be stained first,followed by the p40 antigen. In one aspect, the counterstaining agent ishematoxylin. In one aspect, an annotated image is obtained.

In one aspect, a method for detecting multiple target molecules in abiological sample comprising cells comprises: contacting the biologicalsample with one or more first reagents which generate a detectablesignal in cells comprising a first target molecule; contacting thebiological sample with one or more second reagents which are capable ofgenerating a detectable signal in cells comprising a second targetmolecule under conditions in which the one or more second reagents donot generate a detectable signal; detecting the signal generated by theone or more first reagents; creating conditions in which the one or moresecond reagents generate a signal in cells comprising the second targetmolecule; and detecting the signal generated by the one or more secondreagents.

In one aspect, the method further comprises: obtaining a first digitalimage of the signal generated by the one or more first reagents;obtaining a second digital image of the signal generated by the one ormore first reagents and the signal generated by the one or more secondreagents; and copying a mask of the second digital image to the firstdigital image.

In one aspect, a method for detecting multiple target molecules in abiological sample comprising cells comprises: contacting the biologicalsample with one or more first reagents which generate a detectablesignal in cells comprising a first target molecule; contacting thebiological sample with one or more second reagents which generate adetectable signal in cells comprising a second target molecule, whereinthe detectable signal generated by the one or more second reagents isremovable; detecting the signal generated by the one or more firstreagents and the signal generated by the one or more second reagents;creating conditions in which the signal generated by the one or moresecond reagents is removed; and detecting the signal generated by theone or more first reagents.

In one aspect the method further comprises, obtaining a first digitalimage of the signal generated by the one or more first reagents and thesignal generated by the one or more second reagents; obtaining a seconddigital image of the signal generated by the one or more first reagents;and copying a mask of the first digital image to the second digitalimage.

In one aspect, a method for detecting multiple target molecules in abiological sample comprising cells comprises: contacting the biologicalsample with one or more first reagents which generate a first detectablesignal in cells comprising a first target molecule, wherein said firstdetectable signal is detectable using a first detection method;contacting the biological sample with one or more second reagents whichgenerate a second detectable signal in cells comprising a second targetmolecule, wherein the second detectable signal is detectable using asecond detection method and is substantially undetectable using thefirst detection method and wherein the first detectable signal issubstantially undetectable using the second detection method; detectingthe signal generated by the one or more first reagents using the firstdetection method; and detecting the signal generated by the one or moresecond reagents using the second detection method.

In one aspect, the chromogenic conjugate comprises (a) a chromogenicmoiety, and (b) peroxides substrate moiety, wherein the chromogenicmoieties and the peroxidase substrate moieties are linked together via alinker, and wherein the linker comprises at least one linear chain of atleast 5 consecutively connected atoms. For example, the chromogenicconjugate can be a compound of formula IV.

As described in the examples below, the following compounds may be usedas chromogens, or detectable reagents. These compounds are not intendedto be limiting. The carboxylic acid forms of these structures arecolored and, when converted into their cyclic forms, the compoundsbecome uncolored.

In some aspects, the compound comprises: Formula IV:

In some aspects, the reagent is yellow in color.

In some aspects, the carboxylic acid of Formula IV can have a secondcarbon (i.e., can be CH₂COOH). In some aspects, the ether can be alinker and attaches to a peroxidase substrate moiety.

In some aspects, the compound comprises: Formula VII:

wherein following the conversion to the ring structure of the carboxylicacid, the compound is colorless and does not fluoresce.

In some aspects, the compound comprises: Formula V:

In some aspects, the compound comprises: Formula VI:

In some aspects, the compound comprises: Formula VIII:

II. Sequential Imaging of Biological Samples for Generating TrainingData for Developing Deep Learning Based Models for Image Analysis, forCell Classification, for Feature of Interest Identification, and/or forVirtual Staining of the Biological Sample Training and Applying a DeepLearning Model

The pair of images acquired in one of the methods described above (i.e.,Methods 1 through 3) may be used to train a model that should be able torecognize patterns based on some digitization of antigen B. Twocomputational pipelines have been constructed to exploit the additionallayer of information arriving from the sequential staining process.

The first pipeline (as depicted in FIG. 4A, herein also referred to as“pipeline A”) uses both whole slide image (“WSI”) scans to train aninternal model (“G*”) that takes as input aligned image patches of cells(e.g., pairs of aligned image patches of cells, although not limited topairs) and (in some cases) external labelling, and provides a patchclassification. Model G* can be considered as a ground truth generator(or “GT generator” or the like), and it could be either a deep-learningmodel or based purely on traditional image-processing techniques. The GTgenerator automates the tedious process of assigning each patch itscorrect label. Patches from the original image, containing antigen A,together with labels provided by the GT generator may be used to train acell-classifier deep-learning model (“G”).

The second pipeline (as depicted in FIG. 7A, herein also referred to as“pipeline B”) involves a different intermediate module (i.e.,virtual-stain network (“F”)) that is trained to simulate the antigen Bstaining patterns on a pixel-to-pixel basis. The virtual-stain network Ftakes as an input image stained with the original antigen A and adds avirtual antigen B staining on top of it, as if they were stainedsimultaneously. The virtual-stain layer can then be passed to acell-classifier module (“G*” in pipeline B) that is trained to classifynuclei and/or cells. Module G* can take as an input both the originalstain and the virtual stain (similar to the intermediate module G* ofpipeline A) and (in some cases) external labelling, or can operatesolely with the virtual stain image and (in some cases) externallabelling to provide predictions. The additional intermediate processmay assist in the training process and may provide meaningfulregularization to the process. However, even without scoring and/orclassifying the whole slide, the additional virtual-staining can provideuseful and crucial information to a pathologist and can be used as adecision-support system. An important advantage of the virtual-stainingmodel is that it does not requires a nuclei and/or cell detection module(although, in some cases, it might utilize external labelling as anadditional input) for the training process. It only requires an accuratealignment.

Both pipelines rely on a reliable alignment between the sequentialtissue scans to train the inner modules, and (in some cases) mayeventually a nuclei detection mechanism, to provide overall scoringand/or diagnosis for the whole slide.

In some scenarios, where the staining patterns allow (e.g., nuclearstaining), the process can be automated end to end. In such a case, adigital mask of the localization of antigen B can be done by a suitablemethod that separates the relevant color from the image where thelocation second antigen B is revealed. Following alignment of theuncontaminated first image containing only visible precipitation ofantigen A and the digital mask of antigen B, the digital mask can beconsidered annotation data.

The system, processes, and results are described in detail below withrespect to FIGS. 3 - 9 .

III. Embodiments as Illustrated in the Drawings

We now turn to the embodiments as illustrated by the drawings. FIGS.1-11 illustrate some of the features of tissue staining, for detectablereagents capable of serving as substrates of an enzyme with peroxidaseactivity, and/or for detecting multiple target molecules in biologicalsamples comprising cells, and/or some of the features for implementingannotation data collection and autonomous annotation, and, moreparticularly, to methods, systems, and apparatuses for implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, for feature of interest identification, and/or forvirtual staining of biological samples, as referred to above. Themethods, systems, and apparatuses illustrated by FIGS. 1-11 refer toexamples of different embodiments that include various components andsteps, which can be considered alternatives or which can be used inconjunction with one another in the various embodiments. The descriptionof the illustrated methods, systems, and apparatuses shown in FIGS. 1-11is provided for purposes of illustration and should not be considered tolimit the scope of the different embodiments.

A. Non-Limiting Examples for Tissue Staining Methodology Example 1:Method for Using a Chromogen Visible in Brightfield to Mark the Secondantigen in IHC

The purpose of this method is to run a normal IHC staining to visualizean antigen, and to precipitate an invisible reagent in the tissue, whichcan withstand the process of dehydration and rehydration and be“developed” in a subsequent reaction. Following the initial reactionwhere only one antigen (antigen A) is visible in brightfield due toprecipitation of chromogen in the tissue, the slide in mounted andimaged using a digital slide scanner. The slide is then rehydrated, thesecond antigen (antigen B) is visualized by precipitating a differentchromogen around the secondary antigen, the slide is dehydrated, mountedand imaged. Hence, the first image is a single stain, uncontaminated bythe invisible reagent that is precipitated around the antigen B. In oneaspect, the method may use compounds, such as those described herein,which are colored in their carboxylic acid forms and uncolored or lesscolored when the carboxylic acid cyclizes. In one aspect, this couldalso be Biotin which is invisible, and then visualized with streptavidinHRP and chromogen. Following imaging, the second stain can then be“developed” by precipitating another chromogen visible around the siteof antigen B.

Copying a mask created from the second stain to the first uncontaminatedimage makes it possible to do massive annotation of the uncontaminatedimage of whatever the second stain represents. Following from theexample, it could be used to reveal PD-L1 positive lymphocytes. Inanother investigation, revealing tumor cell nuclei using p40 as antigenB was successfully demonstrated.

TABLE 1 Step Reagent Net Result Target retrieval/dewax Target retrievalbuffer Antigens in tissue are exposed for the reaction Protein blockPeroxidase blocking solution (3% H₂O₂) Endogenous peroxidases areinactivated Mark antigen A Primary antibody A against antigen A AntigenA reacted with antibody A Bind HRP enzymes around antigen A HRP-coupledpolymer that binds primary antibodies Site of antigen A covered withHRP-coupled polymer Precipitate chromogen around antigen A Chromogenwhich is HRP substrate (DAB or HRP Magenta) Chromogen precipitatedaround antigen A and visible in brightfield Denature availableantibodies bound to tissue Sulfuric acid (300 mM) All antibodies aredenatured, but retrieved antigens still available Mark antigen B Primaryantibody B against antigen B Antigen B reacted with antibody B Bind HRPenzymes around antigen B HRP-coupled polymer that binds primaryantibodies Site of antigen B covered with HRP-coupled polymerPrecipitate substrate around antigen B Fluorescein-coupled longsingle-chained polymer (fer-4-flu linker) fer-4-flu linker precipitatedin tissue (invisible in brightfield) Counterstain Hematoxylin Nucleicounterstained Dehydrate and mount Alcohol, xylene, and organic mountingmedium Slide dehydrated and mounted Digital slide scan Remove coverslip,rehydrate Alcohol and xylene Glass and mounting medium removed Bind HRPenzymes around antigen B Anti-FITC antibody coupled to HRP Anti-FITCantibody recognizes Fluorescein molecules on the fer-4-flu linkerPrecipitate chromogen around antigen B Chromogen which is HRP substrate(DAB or HRP Magenta) Chromogen precipitated around antigen B and visiblein brightfield Counterstain Hematoxylin Nuclei counterstained Dehydrateand mount Alcohol, xylene, and organic mounting medium Slide dehydratedand mounted Digital slide scan

(Table 1; Cont.)

Example 2: Method for Using a Dissolvable Chromogen Visible inBrightfield to Mark the second antigen in IHC

The purpose of this method is to run a normal IHC staining with tovisualize an antigen with a non dissolvable chromogen, and toprecipitate a dissolvable chromogen around a second antigen. After theinitial reaction, both antigens A and B are visible and brightfield andcan be mounted with aqueous mounting medium and imaged using a digitalslide scanner. The slide is then incubated in organic solvent (such as96% ethanol) which dissolves the dissolvable chromogen. After this step,the slide is dehydrated, mounted and imaged. Hence, the first image is adouble stain of antigens A and B, whereas the second image contains onlyantigen A. Copying a mask of antigen B created from the first stain tothe second image containing only antigen A makes it possible to domassive annotation of the second image of whatever the stain ofdissolvable chromogen represents. Following from the example mentionedin section A, it could be used to reveal PD-L1 positive lymphocytes.

TABLE 2 Step Reagent Net Result Target retrieval/dewax Target retrievalbuffer Antigens in tissue are exposed for the reaction Protein blockPeroxidase blocking solution (3% H₂O₂) Endogenous peroxidases areinactivated Mark antigen A Primary antibody A against antigen A AntigenA reacted with antibody A Bind HRP enzymes around antigen A HRP-coupledpolymer that binds primary antibodies Site of antigen A covered withHRP-coupled polymer Precipitate chromogen around antigen A Chromogenwhich is HRP substrate (DAB or HRP Magenta) Chromogen precipitatedaround antigen A and visible in brightfield Denature availableantibodies bound to tissue Sulfuric acid (300 mM) All antibodies aredenatured, but retrieved antigens still available Mark antigen B Primaryantibody B against antigen B Antigen B reacted with antibody B Bind HRPenzymes around antigen B HRP-coupled polymer that binds primaryantibodies Site of antigen B covered with HRP-coupled polymerPrecipitate substrate around antigen B Amino ethyl carbazole (“AEC”)Chromogen precipitated around antigen B Counterstain Hematoxylin Nucleicounterstained Mount in aqueous mounting medium Aqueous mounting medium,e.g., faramount Slide mounted Digital slide scan Remove dissolvablechromogen Alcohol or acetone AEC is dissolved Dehydrate and mountAlcohol, xylene, and organic mounting medium Slide dehydrated andmounted Digital slide scan

(Table 2: Cont.)

Example 3: Method for Using a Fluorescence Marker Invisible inBrightfield to Mark The second antigen in IHC

Another method that avoids the mechanical stress associated withmounting and unmounting slides, with resultant tissue shear anddifficulties with the alignment of images to transfer the mask, is toprecipitate a chromogen invisible in brightfield, but visible influorescence during the staining process. This allows for a betteralignment between the first and second image. Steps are shown below:

TABLE 3 Step Reagent Net Result Target retrieval/dewax Target retrievalbuffer Antigens in tissue are exposed for the reaction Protein blockPeroxidase blocking solution (3% H₂O₂) Endogenous peroxidases areinactivated Mark antigen A Primary antibody A against antigen A AntigenA reacted with antibody A Bind HRP enzymes around antigen A HRP-coupledpolymer that binds primary antibodies Site of antigen A covered withHRP-coupled polymer Precipitate chromogen around antigen A Chromogenwhich is HRP substrate (DAB or HRP Magenta) Chromogen precipitatedaround antigen A and visible in brightfield Denature availableantibodies bound to tissue Sulfuric acid (300 mM) All antibodies aredenatured, but retrieved antigens still available Mark antigen B Primaryantibody B against antigen B Antigen B reacted with antibody B Bind HRPenzymes around antigen B HRP-coupled polymer that binds primaryantibodies Site of antigen B covered with HRP-coupled polymerPrecipitate fluorescent substrate around antigen B Rhodamine-basedfluorescent compound coupled to long single-chained polymer Fluorescentcompound precipitated in tissue (invisible in brightfield) CounterstainHematoxylin Nuclei counterstained Dehydrate and mount Alcohol, xylene,and organic mounting medium Slide dehydrated and mounted Digital slidescan using both fluorescence and brightfield

Example 4: Method for Using a Chromogen Visible in Brightfield to Markthe Second target nucleic acid in ISH

The purpose of this method is to run a normal ISH staining to visualizea first target nucleic acid, and to precipitate an invisible reagent inthe tissue, which can be “developed” in a subsequent reaction. Followingthe initial reaction where only one target nucleic acid (target A) isvisible in brightfield due to precipitation of chromogen in the tissue,the slide in mounted and imaged using a digital slide scanner. A secondISH staining is then performed and the second target nucleic acid(target B) is visualized by precipitating a different chromogen aroundthe second target nucleic acid. The slide is then imaged. Hence, thefirst image is a single stain, uncontaminated by the invisible reagentthat is precipitated around target B. Following imaging, the secondstain can then be “developed” by precipitating another chromogen visiblearound the site of target B. It will be appreciated that, rather thanusing precipitation of a chromogen to detect one of the target nucleicacids, a directly labeled nucleic acid probe, such as a nucleic acidprobe having a fluorescent moiety, may be used.

Copying a mask created from the second stain to the first uncontaminatedimage makes it possible to do massive annotation of the uncontaminatedimage of whatever the second stain represents.

TABLE 4 Step Reagent Net Result Dewax Xylene or Clearify Paraffinremoved Hydrate Ethanols Tissue rehydrated Heat pretreatmentPretreatment buffer Probe access improved Enzyme pretreatment Pepsinreagent Probe access improved Dehydrate Ethanols Water removed DenatureNucleic Acids in Tissue Sample and probe A Agilent IQFISH FastHybridization Buffer with probe A Nucleic acids are denatured Hybridizefirst nucleic acid probe to target nucleic acid A where the probe ismodified with moieties which can be observed by fluorescence orrecognized by a primary antibody Agilent IQFISH Fast HybridizationBuffer, Modified nucleic acid probes (for example first nucleic acidprobe modified with CY3) Modified probe hybridized to target nucleicacid A Mark target nucleic acid A First primary antibody againstmodified probe for target nucleic acid A labelled with HRP (for exampleantibody against CY3) First nucleic acid probe bound to first primaryantibody labelled with HRP Precipitate chromogen around target nucleicacid A Chromogen which is HRP substrate (DAB or HRP Magenta) Chromogenprecipitated around antigen A and visible in brightfield Dehydrate andmount Alcohol, xylene and organic mounting medium Slide dehydrated andmounted Digital slide scan Remove coverslip, rehydrate Alcohol andxylene Glass and mounting medium removed Denature antibodies bound totissue Sulfuric acid (300 mM) All antibodies denatured and HRP activityinactivated Dehydrate Ethanols Water removed Denature Nucleic Acids inTissue Sample and probe B Agilent IQFISH Fast Hybridization Buffer withprobe B Nucleic acids are denatured Hybridize second nucleic acid probeto target nucleic acid B where the probe is modified with moieties whichcan be observed by fluorescence or recognized by a primary antibodyAgilent IQFISH Fast Hybridization Buffer, Modified nucleic acid probes(for example, first nucleic acid probe modified with fluorescein)Modified probes hybridized to target nucleic acid B Mark target nucleicacid B Second primary antibody against modified probe for target nucleicacid B labelled with HRP (for example antibody against fluorescein)Second nucleic acid probe bound to second primary antibody labeled withHRP Precipitate substrate around target nucleic acid BFluorescein-coupled long single-chained polymer (fer-4-flu linker)fer-4-flu linker precipitated in tissue (invisible in brightfield)Counterstain Hematoxylin Nuclei counterstained Dehydrate and mountAlcohol, xylene and organic mounting medium Slide dehydrated and mountedDigital slide scan

(Table 4; Cont.)

Example 5: Method for Using a Dissolvable Chromogen Visible inBrightfield to Mark the second target nucleic acid in ISH

The purpose of this method is to run a normal ISH staining to visualizea first target nucleic acid with a non dissolvable chromogen, and toprecipitate a dissolvable chromogen around a second target nucleic acid.After the initial reaction, both target nucleic acids A and B arevisible and brightfield and can be mounted with aqueous mounting mediumand imaged using a digital slide scanner. The slide is then incubated inorganic solvent (such as 96% ethanol) which dissolves the dissolvablechromogen. After this step, the slide is dehydrated, mounted and imaged.Hence, the first image is a double stain of target nucleic acids A andB, whereas the second image contains only target nucleic acid A. It willbe appreciated that, rather than using precipitation of a chromogen todetect one of the target nucleic acids, a directly labeled nucleic acidprobe, such as a nucleic acid probe modified with a fluorescent moiety,may be used.

Copying a mask of target nucleic acid B created from the first stain tothe second image containing only target nucleic acid A makes it possibleto do massive annotation of the second image of whatever the stain ofdissolvable chromogen represents.

TABLE 5 Step Reagent Net Result Dewax Xylene or Clearify Paraffinremoved Hydrate Ethanols Tissue rehydrated Heat pretreatmentPretreatment buffer Probe access improved Enzyme pretreatment Pepsinreagent Probe access improved Dehydrate Ethanols Water removed DenatureNucleic Acids in Tissue Sample and probe A Agilent IQFISH FastHybridization Buffer with probe A Nucleic acids are denatured Hybridizefirst nucleic acid probe to target nucleic acid A where the probe ismodified with a moiety which can be recognized by a first primaryantibody Agilent IQFISH Fast Hybridization Buffer, Modified nucleic acidprobe (for example, nucleic acid probe modified with fluorescein)Modified probe hybridized to target nucleic acid A Mark target nucleicacid A First primary antibody against modified probe for target nucleicacid A labelled with HRP (for example, antibody against fluorescein)First nucleic acid probe bound to first primary antibody labelled withHRP Precipitate chromogen around target nucleic acid A Chromogen whichis HRP substrate (DAB or HRP Magenta) Chromogen precipitated aroundtarget nucleic acid A and visible in brightfield Denature antibodiesbound to tissue Sulfuric acid (300 mM) All antibodies denatured and HRPactivity inactivated Dehydrate Ethanols Water removed Denature NucleicAcids in Tissue Sample and probe B Agilent IQFISH Fast HybridizationBuffer with probe B Nucleic acids and probe denatured Hybridize secondnucleic acid probe to target nucleic acid B where the probe is modifiedwith a moiety which can be recognized by a second primary antibodyAgilent IQFISH Fast Hybridization Buffer, Modified nucleic acid probe B(for example, nucleic acid probe modified with CY3) Modified probehybridized to target nucleic acid B Mark target nucleic acid B Secondprimary antibody against modified probe for target nucleic acid Blabelled with HRP (for example, antibody against CY3) Second nucleicacid probe bound to second primary antibody labelled with HRPPrecipitate substrate around target nucleic acid B Amino ethyl carbazole(AEC) Chromogen precipitated around target nucleic acid B CounterstainHematoxylin Nuclei counterstained Mount in aqueous mounting mediumAqueous mounting medium e.g. faramount Slide mounted Digital slide scanRemove dissolved chromogen Alcohol or acetone AEC is dissolved Dehydrateand mount Alcohol, xylene, and organic mounting medium Slide dehydratedand mounted Digital slide scan

(Table 5; Cont.)

Example 6: Method for Using a Chromogen Visible in Brightfield to Markthe First marker in IHC and a fluorescence marker invisible inbrightfield to mark the second target nucleic acid in ISH

Another option is to precipitate a chromogen invisible in brightfield,but visible in fluorescence during the staining process in combinationwith an IHC reaction to precipitate a chromogen visible in brightfieldaround a certain marker. It will be appreciated that, rather than usingprecipitation of a chromogen to detect one of the target nucleic acids,a directly labeled nucleic acid probe, such as a nucleic acid probemodified with a fluorescent moiety, may be used. Steps are shown below:

TABLE 6 Step Reagent Net Result Denature Nucleic Acids in Tissue SampleAgilent IQFISH Fast Hybridization Buffer Nucleic acids are denaturedHybridize first nucleic acid probe to target nucleic acid A where theprobe is modified with a moiety which can be recognized by a firstprimary antibody Agilent IQFISH Fast Hybridization Buffer, Modifiednucleic acid probe Modified probe hybridized to target nucleic acid AMark target nucleic acid A First primary antibody against modified probefor target nucleic acid A First nucleic acid probe bound to firstprimary antibody Bind HRP enzymes around target nucleic acid AHRP-coupled polymer that binds primary antibodies Site of target nucleicacid A covered with HRP-coupled polymer Precipitate chromogen aroundtarget nucleic acid A Chromogen which is HRP substrate (DAB or HRPMagenta) Chromogen precipitated around target nucleic acid A and visiblein brightfield Denature first nucleic acid probe bound to target nucleicacid A, remove probe with wash procedure Agilent IQFISH FastHybridization Buffer First nucleic acid probe removed Hybridize secondnucleic acid probe to target nucleic acid B where the probe is modifiedwith a moiety which can be recognized by a second primary antibodyAgilent IQFISH Fast Hybridization Buffer, Modified nucleic acid probeModified probe hybridized to target nucleic acid B Digital slide scanTarget retrieval/dewax Target retrieval buffer Antigens in tissue areexposed for the reaction Protein block Peroxidase blocking solution (3%H₂O₂) Endogenous peroxidases are inactivated Mark antigen B Primaryantibody B against antigen B Antigen B reacted with antibody B Bind HRPenzymes around antigen B HRP-coupled polymer that binds primaryantibodies Site of antigen B covered with HRP-coupled polymerPrecipitate chromogen around antigen B Chromogen which is HRP substrate(DAB or HRP Magenta) Chromogen precipitated around antigen B and visiblein brightfield) Counterstain Hematoxylin Nuclei counterstained Digitalslide scan

(Table 6; Cont.)

Example 7: Method for Doing a Dissolvable Primary Stain Visible inBrightfield, followed by a second reaction to mark an antigen in IHC

The purpose of this method is to do a primary stain (such as H&E),followed by dissolution of this stain, imaging, followed by a normal IHCstaining with to visualize an antigen with a non-dissolvable chromogen.After the initial reaction, the primary stain is visible. The slide isthen incubated in organic solvent (such as 96% ethanol) which dissolvesthe dissolvable primary stain. After this step, the slide is dehydrated,mounted, and imaged. After this, a normal IHC reaction is run to mark acertain antigen, followed by mounting and imaging. The mask from the IHCstain can then be applied to the primary stain to reveal cell identity.

TABLE 7 Step Reagent Net Result Target retrieval/dewax Target retrievalbuffer Antigens in tissue are exposed for the reaction Primary stainHematoxylin and Eosin Primary stain completed Mount in aqueous mountingmedium Aqueous mounting medium e.g. faramount Slide mounted Digitalslide scan Protein block Peroxidase blocking solution (3% H₂O₂)Endogenous peroxidases are inactivated Mark antigen A Primary antibody Aagainst antigen A Antigen A reacted with antibody A Bind HRP enzymesaround antigen A HRP-coupled polymer that binds primary antibodies Siteof antigen A covered with HRP-coupled polymer Precipitate chromogenaround antigen A Chromogen which is HRP substrate (DAB or HRP Magenta)Chromogen precipitated around antigen A and visible in brightfieldCounterstain Hematoxylin Nuclei counterstained Dehydrate and mountAlcohol, xylene, and organic mounting medium Slide dehydrated andmounted Digital slide scan

Example 8:

In one experiment, PD-L1 was used. A notorious problem with PD-L1 stainsare PD-L1 positive lymphocytes which are very hard to distinguish fromtumor cells. Hence, marking several types of lymphocytes using acomputational approach is challenging. Adenocarcinomas of the lung aswell as squamous carcinomas of the lung were stained, first with PD-L1with DAB chromogen as well as an invisible tag, fer4flu around alymphocyte/macrophage marker in the same reaction. The tissue wasstained for CD3 (T-cells/NK cells), CD20 (B-cells) and CD68(macrophages). An invisible tag was used because the process of organicmounting made it very difficult to just do a whole new reaction fromscratch. The antigens are very severely affected by the process oforganic mounting, whereas this is not the case for aqueous mounting.This is important, because the image used for prediction has to be as itwill be in the clinic - and here, it will almost always be permanentlymounted in organic mounting medium. Aqueous mounting media look verydifferent, have different refractive indices. This allows for anuncontaminated first image in organic mounting medium. The result ofthis experiment (following optimization), was that it was possible to dothis workflow. In all the cases stained, the sequential imaging andstaining for the said markers was done. The magenta staining was thenused a training mask to create a virtual prediction of the staining ofeach marker, only based on the PD-L1 stain. See FIG. 1A showing afterthe first stain - PD-L1 visible precipitated with DAB. Invisible fer4flulinker also precipitated at sites of CD68 antigen. Slide is mounted inorganic mounting medium and imaged. FIG. 1B shows Development of fer4flubound at CD68 positive sites with HRP magenta.

For p40, adenocarcinomas of the lung was studied. A permanently mounteduncontaminated first image was obtained through precipitating thefer4flu linker around the p40 sites, mounted in organic medium, imaged,removed coverslip and developed the p40 stain using HRP magenta. SeeFIG. 2A showing after first stain. Fer4flu linker is precipitated aroundp40 sites. Slide is mounted in organic mounting medium and imaged. FIG.2B shows p40 positivity is developed using HRP Magenta at sites offer4flu linker.

In both cases, the proper positive and negative controls ensured thatthe quality of the method is sufficient.

Example 9:

High affinity antibodies, which ensure robust staining reactions, cansometime be very difficult to denature using heat or harsh conditions. Amethod of obtaining effective antibody stripping between reactions hastherefore been designed, as described in Table 8 below. The designedmethod was also shown be an effective way to remove plastic coverslipswithout compromising subsequent reaction (see, steps 5-12 in the belowTable 8), a process which can be used in different settings.

TABLE 8 Step Action 1 IHC reaction: Stain A visualized by precipitationof an HRP substrate (FL or BF) 2 Dehydrate in gradients of alcohol andxylene 3 Mounting using Sakura TissueTek resin-covered plastic tape(xylene is added to dissolve tape into tissue) 4 Image in brightfieldscan 5 Remove plastic tape start 6 Incubate for 5-7 minutes in 100%acetone, covering the slides completely 7 Plastic tape is loosed, removeusing pincers 8 Move slides to fresh 100% acetone bath with magnetstirrer at 1000 rpms to dissolve remaining plastic 9 Move slides to 96%ethanol for 2 minutes 10 Move slides to 96% ethanol for 2 minutes 11Move slides to 70% ethanol for 2 minutes 12 Move slides 1X RT Dako Washbuffer for 5 minutes 13 Preheat vertical containers with a strippingbuffer* to exactly 56° C. in closed, shaking water-bath in fume hood 14Incubate slides for 30 minutes at 56° C. with shaking 15 Transfer to 1XRT Dako wash buffer; 5 minutes times 3 (change buffer every 5 minutesfor 15 minutes) 16 Transfer to 1X RT Dako wash buffer; 5 minutes times 3(change buffer every 15 minutes for 45 minutes) 17 Ready for next IHCreaction (precipate HPR substrate around antigen B) ^(∗)Stripping bufferwas prepared as follows: 200 ml 1X (prepare and refrigerate without the2-ME) 20 ml SDS 10% 12.5 ml Tris HCl pH 6.8 (0.5 M) 167,5 ml DistilledH₂O

Before use, add 0.8 ml 2-mercaptoethanol; once added, the half-life isabout 100 hrs.

Work under a fume hood. Scale up or down the volumes and chemicalsaccordingly.

Example 10 Dark Quenchers

It has been observed that DAB was a potent dark quencher for fluorescentexcitations and emissions. Dark quenchers are molecules that, by virtualof their physiochemical properties, absorb light of certain wavelengths,which are converted to heat.

A method that is usable in certain types of multiplexing or sequentialstaining experiments was designed. Since high-affinity antibodies arehard to elute after binding, especially because HRP substrates becomecovalently attached to adjencent molecules by process of free radicalgeneration, there are significant challenges of cross-reactivity betweenreactions when multiple stainings are applied. For example, if amouse-anti-target1 antibody is applied, visualized with an HRP-polymerand HRP substrate 1, followed by enzyme deactivation (with sulfuric acidfor example), and a different mouse-anti-target2 antibody is applied,and visualized that with HRP substrate 2, there may be some crossoverfor the visualization system between target 1 and target 2, depending onthe sensitivity of these HRP substrates.

DAB was therefore used to quench the signal from a previous reaction byexploiting the crossover between reactions, as depicted in Table 9 andin FIG. 19 .

TABLE 9 Step Description 1 Do PharmDx stain (e.g. PDL1 with DAB) 2Mount, Scan in BF, unmount and clear counterstain (hematoxylin) 3 Domouse-anti human stain n1 (for instance CD68) with fer4flu as visualizer4 Mount, Scan in FL, unmount 5 Incubate with secondary reagent(goat-anti mouse-HRP conjugated polymer) and precipitate DAB. Stain n1is now hidden in FL mode and is not visualized even if more fer4flu isprecipiated 6 Do mouse-anti human stain n2 (for instance CD20) withfer4flu as visualizer 7 Mount, scan in FL, unmount 8 Cycle back betweensteps 5 and 8.

Using the described methodology allows using the same fluorochromes forsequential reactions without crossover.

FIG. 20 presents an exemplary image of a tissue section where multipleround of staining has been done using DAB as a quencher betweenreactions. First PDL1 (left), then CD8 for cytotoxic T-cells (center)which was subsequently dark quenched and stained for B-cells (right).

Example 11: RNA-CISH / IHC Multiplex

Information about RNA transcripts was used in determining cell identity.This was achived with sequential reactions of first using an RNA CISHkit from Bioteche (RNAscope) for a LAG3 transcript, which was visualisedusing a dissolvable chromogen (AEC). Following this, another reactionwas added on top with LAG3 IHC (unreleased clone, internal development)and DAB using the normal IHC protocol. Thus, on the brightfield image,there is the RNAscope mask for which cells had RNA transcripts.

Example 12:

Specific antibody use for labeling nuclear periphery in sequentialreactions A deep learning based nuclear classifier was built by labelingnuclei centroids as well as the nuclear periphery, in order to predictthe two surfaces and combine that information to generate a nuclearmask. FIG. 21 presents an immunofluorescence image of an antibodyagainst LaminB1, and a nuclear envelope protein which effectively stainsthe nuclear periphery (Coumaric acid-L42-Rhodamine101, different fromfer4flu-LaminB1).

Such stains improve the nuclear classifier, by using large-scalebiological annotations to create annotations for the nuclear periphery.

B. Non-Limiting Examples for Sequential Imaging of Biological Samplesfor Generating Training Data for Developing Deep Learning Based Modelsfor Image Analysis, for Cell Classification, for Feature of InterestIdentification, and/or for Virtual Staining of the Biological Sample

With reference to the figures, FIG. 3 is a schematic diagramillustrating a system 300 for implementing sequential imaging ofbiological samples for generating training data for developing deeplearning based models for image analysis, for cell classification,and/or for features of interest identification of biological samples, inaccordance with various embodiments.

In the non-limiting embodiment of FIG. 3 , system 300 may comprise acomputing system 305 a, an artificial intelligence (“AI”) system 305 d(optional), and a data store or database 310 a that is local to thecomputing system 305 a and/or the AI system 305 d. In some cases, thedatabase 310 a may be external, yet communicatively coupled, to thecomputing system 305 a. In other cases, the database 310 a may beintegrated within the computing system 305 a. In some embodiments, theAI system 305 d - which may include, but is not limited to, at least oneof a machine learning system, a deep learning system, a modelarchitecture, a statistical model-based system, or a deterministicanalysis system, and/or the like - may be external, yet communicativelycoupled, to the computing system 305 a or may be integrated within thecomputing system 305 a. In some instances, the model architecture maycomprise at least one of a neural network, a convolutional neuralnetwork (“CNN”), or a fully convolutional network (“FCN”) (which mayinclude a U-Net framework or the like), and/or the like.

System 300, according to some embodiments, may further comprise adisplay device 320 that may allow a user 325 to view a field of view(“FOV”) of a biological sample or an image(s) or video(s) of thebiological sample. System 300 may further comprise one or more userdevices 340 (optional), one or more audio sensors 335 (optional), acamera(s) 330 (optional), and a microscope 315. In some instances, theone or more user devices 340 may include, without limitation, smartphones, mobile phones, tablet computers, laptop computers, desktopcomputers, keyboards, keypads, computer mice, or monitors, and/or thelike. In some cases, the one or more audio sensors 335 may include, butare not limited to, one or more microphones, one or more voicerecorders, or one or more audio recorders, and/or the like. In someinstances, the camera 330 may include, without limitation, one or moreeye tracking sensors, one or more motion sensors, or one or moretracking sensors, and/or the like.

According to some embodiments, the one or more user devices 340 may beused to receive user input from the user 325 indicative of annotationsor labeling of objects of interest observed by the user 325 whileviewing the field of view of the biological sample, whether viewing on adisplay screen of the display device 320 or viewing through aneyepiece(s) of the microscope 315.The one or more audio sensors 335 maybe used to record vocal or spoken annotations by the user 325 while theuser 325 is viewing the FOV 330 b of the biological sample either on thedisplay device 320 or through the eyepiece(s) of the microscope 315. Thecamera 330 may capture images or videos of the user 325 (in some cases,capturing images or videos of at least one eye of the user 325) whilethe user 325 is within the FOV 330 a of camera 330. The features of gazetracking and voice annotation are described in greater detail in the‘105 Application, which has already been incorporated by reference inits entirety for all purposes.

Computing system 305 a may communicatively couple (either via wireless(as depicted by lightning bolt symbols, or the like) or wired connection(as depicted by connecting lines)) with one or more of the AI system 305d, the database(s) 310 a, the display device 320, the one or more userdevices 340, the one or more audio sensors 335, the camera 330, and/orthe microscope 315. Computing system 305 a, the AI system 305 d, thedatabase(s) 310 a, the display device 320, the one or more user devices340, the one or more audio sensors 335, the camera 330, and/or themicroscope 315 may be disposed or located within work environment 345,which may include, but is not limited to, one of a laboratory, a clinic,a medical facility, a research facility, a healthcare facility, or aroom, and/or the like.

System 300 may further comprise remote computing system 305 b(optional), AI system 305 c (optional), and database(s) 310b (optional)that may communicatively couple with computing system 305 a and/or AIsystem 305 d via network(s) 350. In some cases, the remote computingsystem 305 b may include, but is not limited to, a web server, a webbrowser, or a cloud computing system, and/or the like. Remote computingsystem 305 b, AI system 305 c, and database(s) 310 b may otherwise besimilar, if not identical, to computing system 305 a, the AI system 305d, and the database(s) 310 a, respectively.

Merely by way of example, network(s) 350 may each include a local areanetwork (“LAN”), including, without limitation, a fiber network, anEthernet network, a Token-Ring® network, and/or the like; a wide-areanetwork (“WAN”); a wireless wide area network (“WWAN”); a virtualnetwork, such as a virtual private network (“VPN”); the Internet; anintranet; an extranet; a public switched telephone network (“PSTN”); aninfra-red network; a wireless network, including, without limitation, anetwork operating under any of the IEEE 802.11 suite of protocols, theBluetooth® protocol known in the art, and/or any other wirelessprotocol; and/or any combination of these and/or other networks. In aparticular embodiment, network(s) 350 may each include an access networkof an Internet service provider (“ISP”). In another embodiment,network(s) 350 may each include a core network of the ISP, and/or theInternet.

In operation, computing system 305 a or 305 b and/or AI system 305 d or305 c (collectively, “computing system”) may receive a first image of afirst biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample that has been stained with afirst stain; and may receive a second image of the first biologicalsample, the second image comprising a second FOV of the first biologicalsample that has been stained with a second stain. The computing systemmay autonomously create a first set of image patches based on the firstimage and the second image, by extracting a portion of the first imageand extracting a corresponding portion of the second image, the firstset of image patches comprising a first patch corresponding to theextracted portion of the first image and a second patch corresponding tothe extracted portion of the second image. In such cases, the first setof image patches may comprise labeling of instances of features ofinterest in the first biological sample that is based at least in parton at least one of information contained in the first patch, informationcontained in the second patch, or information contained in one or moreexternal labeling sources.

The computing system may utilize an AI system (e.g., AI system 305 d or305 c, or the like) to train a first model (“Model G*” or “(AI) ModelG*”; which may be an AI model or can, instead, be either a statisticalmodel or a deterministic algorithm model, or the like) to generate firstinstance classification of features of interest (“Ground Truth”) in thefirst biological sample, based at least in part on the first set ofimage patches and the labeling of instances of features of interestcontained in the first set of image patches; and may utilize the AIsystem to train a second AI model (“Model G”) to identify instances offeatures of interest in the first biological sample, based at least inpart on the first patch and the first instance classification offeatures of interest generated by Model G*.

According to some embodiments, the first biological sample may include,without limitation, one of a human tissue sample, an animal tissuesample, a plant tissue sample, or an artificially produced tissuesample, and/or the like. In some cases, the artificially produced tissuesample may include, but is not limited to, at least one of an artificiallab-grown tissue sample, an artificial commercially produced tissuesample, an artificially produced heart valve tissue sample, anartificially produced organ tissue sample, a 3D printed tissue sample, a3D produced cell culture, and/or the like. In some instances, thefeatures of interest may include, but are not limited to, at least oneof normal cells, abnormal cells, damaged cells, cancer cells, tumors,subcellular structures, or organ structures, and/or the like.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample (including, but notlimited to, at least one of first antigens, first nuclei, first cellwalls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. Similarly, the second image maycomprise one of highlighting of second features of interest in the firstbiological sample (including, but not limited to, at least one of secondantigens, second nuclei, second cell walls, second cell structures,second antibodies, second normal cells, second abnormal cells, seconddamaged cells, second cancer cells, second tumors, second subcellularstructures, second organ structures, or other features of interest, orthe like) by the second stain that had been applied to the firstbiological sample in addition to highlighting of the first features ofinterest by the first stain or highlighting of the second features ofinterest by the second stain without highlighting of the first featuresof interest by the first stain, the second features of interest beingdifferent from the first features of interest. In some cases, the secondstain may be one of the same as the first stain but used to stain thesecond features of interest or different from the first stain, or thelike.

Merely by way of example, in some cases, the first stain may include,without limitation, at least one of Hematoxylin, Acridine orange,Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystal violet,4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker (which may resultfrom the use of imaging techniques including, but not limited to, Ramanspectroscopy, near infrared (“NIR”) spectroscopy, autofluorescenceimaging, or phase imaging, and/or the like, and which may be used tohighlight features of interest without an external dye, or the like),and/or the like. In some cases, the contrast when using label-freeimaging techniques may be generated without additional markers such asfluorescent dyes or chromogen dyes, or the like. Likewise, the secondstain may include, but is not limited to, at least one of Hematoxylin,Acridine orange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet,Crystal violet, DAPI, Eosin, Ethidium bromide intercalates, Acidfuchsine, Hoechst stain, Iodine, Malachite green, Methyl green,Methylene blue, Neutral red, Nile blue, Nile red, Osmium tetroxide,Propidium Iodide, Rhodamine, Safranine, PD-L1 stain, DAB chromogen,Magenta chromogen, cyanine chromogen, CD3 stain, CD20 stain, CD68 stain,p40 stain, antibody-based stain, or label-free imaging marker, and/orthe like.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. “Color or brightfield image(s)” may refer to either acolor image(s) each resulting from the use of one or a combination orappropriate color sensors or color filters (in some cases, embodied byan RGB image, but is not limited to red, green, and/or blue coloredimages) or a brightfield image(s) each resulting from the use of a lightsource(s); for simplicity, these terms are intended herein to beinterchangeable. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image (alsoreferred to as “intrinsic fluorescence image” or the like) or a firstlabelled fluorescence image (also referred to as “extrinsic fluorescenceimage” or the like) having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. Similarly, thesecond image may include, without limitation, one of a second set ofcolor or brightfield images, a second fluorescence image, a second phaseimage, or a second spectral image, and/or the like, where the second setof color or brightfield images may include, but is not limited to, asecond R image, a second G image, and a second B image, and/or the like.The second fluorescence image may include, but is not limited to, atleast one of second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The second spectral imagemay include, but is not limited to, at least one of a second Ramanspectroscopy image, a second NIR spectroscopy image, a secondmultispectral image, a second hyperspectral image, or a second fullspectral image, and/or the like.

In some embodiments, the computing system may align the first image withthe second image to create first set of aligned images, by aligning oneor more features of interest in the first biological sample as depictedin the first image with the same one or more features of interest in thefirst biological sample as depicted in the second image. In someinstances, aligning the first image with the second image may be one ofperformed manually using manual inputs to the computing system orperformed autonomously, where autonomous alignment may comprisealignment using at least one of automated global alignment techniques oroptical flow alignment techniques, or the like. In some cases, the firstset of aligned images may comprise a third FOV of the first biologicalsample, the third FOV being either different from each of the first FOVof the first image and the second FOV of the second image, or the sameas one of the first FOV or the second FOV. Herein, FOV may refer to aregion of the biological sample as captured in the first image or thesecond image. Prior to alignment (assuming alignment is necessary), the(first) FOV of the first image, although overlapping with the (second)FOV of the second image, may be slightly different from the FOV of thesecond image. The aligned image is intended to depict the FOV of thearea of interest in the image of the first biological sample, and thusmay be referred to as the intended, desired, or target FOV. In the casethat alignment is not needed (either where the first and second imagesare pre-aligned or where the process that produces the first and secondimages results in automatically aligned images), the first FOV and thesecond FOV would be identical.

According to some embodiments, the computing system may receive a thirdimage of the first biological sample, the third image comprising a thirdFOV of the first biological sample that has been stained with a thirdstain different from each of the first stain and the second stain; andmay autonomously perform one of: aligning the third image with each ofthe first image and the second image to create second aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the third image with the same one or more features ofinterest in the first biological sample as depicted in each of the firstimage and the second image; or aligning the third image with the firstaligned images to create second aligned images, by aligning one or morefeatures of interest in the first biological sample as depicted in thethird image with the same one or more features of interest in the firstbiological sample as depicted in the first aligned images.

The computing system may partially or fully autonomously create secondaligned image patches from the second aligned images, by extracting aportion of the second aligned images, the portion of the second alignedimages comprising a first patch corresponding to the extracted portionof the first image, a second patch corresponding to the extractedportion of the second image, and a third patch corresponding to theextracted portion of the third image; may train the AI system to updateModel G* to generate second instance classification of features ofinterest in the first biological sample, based at least in part on thesecond aligned image patches and the labeling of instances of featuresof interest contained in the first set of image patches; and may trainthe AI system to update Model G to identify instances of features ofinterest in the first biological sample, based at least in part on thefirst patch and the second instance classification of features ofinterest generated by Model G*.

In some embodiments, the first image may comprise highlighting ordesignation of first features of interest in the first biological sample(including, but not limited to, at least one of first antigens, firstnuclei, first cell walls, first cell structures, first antibodies, firstnormal cells, first abnormal cells, first damaged cells, first cancercells, first tumors, first subcellular structures, first organstructures, or other features of interest, or the like) by the firststain that had been applied to the first biological sample. In somecases, the second image may comprise one of highlighting of secondfeatures of interest in the first biological sample (including, but notlimited to, at least one of second antigens, second nuclei, second cellwalls, second cell structures, second antibodies, second normal cells,second abnormal cells, second damaged cells, second cancer cells, secondtumors, second subcellular structures, second organ structures, or otherfeatures of interest, or the like) by the second stain that had beenapplied to the first biological sample in addition to highlighting ordesignation of the first features of interest by the first stain orhighlighting of the second features of interest by the second stainwithout highlighting of the first features of interest by the firststain, the second features of interest being different from the firstfeatures of interest. In some instances, the third image may compriseone of highlighting of third features of interest in the firstbiological sample (including, but not limited to, at least one of thirdantigens, third nuclei, third cell walls, third cell structures, thirdantibodies, third normal cells, third abnormal cells, third damagedcells, third cancer cells, third tumors, third subcellular structures,third organ structures, or other features of interest, or the like) bythe third stain that had been applied to the first biological sample inaddition to highlighting of the first features of interest by the firststain and highlighting of the second features of interest by the secondstain, highlighting of the third features of interest by the third stainin addition to only one of highlighting of the first features ofinterest by the first stain or highlighting of the second features ofinterest by the second stain, or highlighting of the third features ofinterest by the third stain without any of highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain, and/or the like, the thirdfeatures of interest being different from each of the first features ofinterest and the second features of interest.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image or a firstlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. Likewise, thesecond image may include, without limitation, one of a second set ofcolor or brightfield images, a second fluorescence image, a second phaseimage, or a second spectral image, and/or the like, where the second setof color or brightfield images may include, but is not limited to, asecond R image, a second G image, and a second B image, and/or the like.The second fluorescence image may include, but is not limited to, atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The second spectral imagemay include, but is not limited to, at least one of a second Ramanspectroscopy image, a second NIR spectroscopy image, a secondmultispectral image, a second hyperspectral image, or a second fullspectral image, and/or the like. Similarly, the third image may include,without limitation, one of a third set of color or brightfield images, athird fluorescence image, a third phase image, or a third spectralimage, and/or the like, where the third set of color or brightfieldimages may include, but is not limited to, a third R image, a third Gimage, and a third B image, and/or the like. The third fluorescenceimage may include, but is not limited to, at least one of a thirdautofluorescence image or a third labelled fluorescence image having oneor more channels with different excitation or emission characteristics,and/or the like. The third spectral image may include, but is notlimited to, at least one of a third Raman spectroscopy image, a thirdNIR spectroscopy image, a third multispectral image, a thirdhyperspectral image, or a third full spectral image, and/or the like.

In some embodiments, the computing system may receive a fourth image,the fourth image comprising one of a fourth FOV of the first biologicalsample different from the first FOV and the second FOV or a fifth FOV ofa second biological sample, where the second biological sample may bedifferent from the first biological sample; and may identify, usingModel G, second instances of features of interest in the secondbiological sample, based at least in part on the fourth image and basedat least in part on training of Model G using the first patch and thefirst instance classification of features of interest generated by ModelG*. In some cases, the fourth image may comprise the fifth FOV and mayfurther comprise only highlighting of first features of interest in thesecond biological sample by the first stain that had been applied to thesecond biological sample. According to some embodiments, the computingsystem may generate, using Model G, a clinical score, based at least inpart on the identified second instances of features of interest.

In another aspect, the computing system may receive a first image of afirst biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample; and may identify, using afirst AI model (“Model G”) that is generated or updated by a trained AIsystem, first instances of features of interest in the first biologicalsample, based at least in part on the first image and based at least inpart on training of Model G using a first patch and first instanceclassification of features of interest generated by a second model(Model G*) that is generated or updated by the trained AI system byusing first aligned image patches and labeling of instances of featuresof interest contained in the first patch. In some cases, the firstaligned image patches may comprise an extracted portion of first alignedimages, which may comprise a second image and a third image that havebeen aligned. The second image may comprise a second FOV of a secondbiological sample that has been stained with a first stain, the secondbiological sample being different from the first biological sample. Thethird image may comprise a third FOV of the second biological samplethat has been stained with a second stain. The second patch may compriselabeling of instances of features of interest as shown in the extractedportion of the second image of the first aligned images. The first imagemay comprise highlighting of first features of interest in the secondbiological sample (including, but not limited to, at least one of firstantigens, first nuclei, first cell walls, first cell structures, firstantibodies, first normal cells, first abnormal cells, first damagedcells, first cancer cells, first tumors, first subcellular structures,first organ structures, or other features of interest, or the like) bythe first stain that had been applied to the second biological sample.The second patch may comprise one of highlighting of second features ofinterest in the second biological sample (including, but not limited to,at least one of second antigens, second nuclei, second cell walls,second cell structures, second antibodies, second normal cells, secondabnormal cells, second damaged cells, second cancer cells, secondtumors, second subcellular structures, second organ structures, or otherfeatures of interest, or the like) by the second stain that had beenapplied to the second biological sample in addition to highlighting ofthe first features of interest by the first stain or highlighting of thesecond features of interest by the second stain without highlighting ofthe first features of interest by the first stain.

In yet another aspect, the computing system may receive first instanceclassification of features of interest (“Ground Truth”) in a firstbiological sample that has been sequentially stained, the Ground Truthhaving been generated by a trained first model (“Model G*”) that hasbeen trained or updated by an AI system, wherein the Ground Truth isgenerated by using first aligned image patches and labeling of instancesof features of interest contained in the first aligned image patches,wherein the first aligned image patches comprise an extracted portion offirst aligned images, wherein the first aligned images comprise a firstimage and a second image that have been aligned, wherein the first imagecomprises a first FOV of the first biological sample that has beenstained with a first stain, wherein the second image comprises a secondFOV of the first biological sample that has been stained with a secondstain, wherein the first aligned image patches comprise labeling ofinstances of features of interest as shown in the extracted portion ofthe first aligned images; and may utilize the AI system to train asecond AI model (“Model G”) to identify instances of features ofinterest in the first biological sample, based at least in part on thefirst instance classification of features of interest generated by ModelG*.

In still another aspect, the computing system may generate ground truthfor developing accurate AI models for biological image interpretation,based at least in part on images of a first biological sample depictingsequential staining of the first biological sample.

In an aspect, the computing system may receive a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample that has been stained with afirst stain; may receive a second image of the first biological sample,the second image comprising a second FOV of the first biological samplethat has been stained with at least a second stain; and may align thefirst image with the second image to create first aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the first image with the same one or more features ofinterest in the first biological sample as depicted in the second image.

The computing system may autonomously create first aligned image patchesfrom the first aligned images, by extracting a portion of the firstaligned images, the portion of the first aligned images comprising afirst patch corresponding to the extracted portion of the first imageand a second patch corresponding to the extracted portion of the secondimage; may utilize an AI system to train a first AI model (“Model F”) togenerate a third patch comprising a virtual stain of the first alignedimage patches, based at least in part on the first patch and the secondpatch, the virtual stain simulating staining by at least the secondstain of features of interest in the first biological sample as shown inthe second patch; and may utilize the AI system to train a second model(“Model G*”) to identify or classify first instances of features ofinterest in the first biological sample, based at least in part on thethird patch and based at least in part on results from an externalinstance classification process or a region of interest detectionprocess.

According to some embodiments, the first image may comprise highlightingof first features of interest in the first biological sample (including,but not limited to, at least one of first antigens, first nuclei, firstcell walls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. In some cases, the third patchmay comprise one of highlighting of the first features of interest bythe first stain and highlighting of second features of interest in thefirst biological sample (including, but not limited to, at least one ofsecond antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the virtual stain that simulates the secondstain having been applied to the first biological sample or highlightingof the first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest. In someinstances, the second stain may be one of the same as the first stainbut used to stain the second features of interest or different from thefirst stain. In some embodiments, the second FOV of the first biologicalsample that has been stained with at least the second stain may compriseone of: (a) an image or FOV of sequentially staining of the firstbiological sample with two or more stains, the two or more stainscomprising the first stain and the at least the second stain; (b) animage or FOV of fluorescent staining of the first biological sample withthe second stain without the first stain; (c) an image or FOV of singlechannel grayscale staining of the first biological sample; or (d) animage or FOV of fluorescent, single channel, grayscale staining of thefirst biological sample; and/or the like.

Merely by way of example, in some cases, the first stain may include,without limitation, at least one of Hematoxylin, Acridine orange,Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystal violet,4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.Similarly, the second stain may include, but is not limited to, at leastone of Hematoxylin, Acridine orange, Bismarck brown, Carmine, Coomassieblue, Cresyl violet, Crystal violet, DAPI, Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, PD-L1 stain, DABchromogen, Magenta chromogen, cyanine chromogen, CD3 stain, CD20 stain,CD68 stain, p40 stain, antibody-based stain, or label-free imagingmarker, and/or the like.

In some embodiments, the first image may include, without limitation,one of a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like,where the first set of color or brightfield images may include, but isnot limited to, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. In some instances, the second image may include,without limitation, one of a second set of color or brightfield images,a second fluorescence image, a second phase image, or a second spectralimage, and/or the like, where the second set of color or brightfieldimages may include, but is not limited to, a second R image, a second Gimage, and a second B image, and/or the like. The second fluorescenceimage may include, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike. In some cases, the third patch may include, without limitation,one of a third set of color or brightfield images, a third fluorescenceimage, a third phase image, or a third spectral image, and/or the like,where the third set of color or brightfield images may include, but isnot limited to, a third R image, a third G image, and a third B image,and/or the like. The third fluorescence image may include, but is notlimited to, at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may include, but is not limited to, at least one of athird Raman spectroscopy image, a third NIR spectroscopy image, a thirdmultispectral image, a third hyperspectral image, or a third fullspectral image, and/or the like.

According to some embodiments, aligning the first image with the secondimage may be one of performed manually using manual inputs to thecomputing system or performed autonomously, where autonomous alignmentmay comprise alignment using at least one of automated global alignmenttechniques or optical flow alignment techniques.

In some embodiments, training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample may comprise training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample, based at least in part on one or more of the first patch, thesecond patch, or the third patch and based at least in part on theresults from the external instance classification process or the regionof interest detection process. In some cases, the external instanceclassification process or the region of interest detection process mayeach include, without limitation, at least one of detection of nuclei inthe first image or the first patch by a nuclei detection method,identification of nuclei in the first image or the first patch by apathologist, detection of features of interest in the first image or thefirst patch by a feature detection method, or identification of featuresof interest in the first image or the first patch by the pathologist,and/or the like.

According to some embodiments, training the AI system to update Model Fmay comprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may include,but is not limited to, one of a mean squared error loss function, a meansquared logarithmic error loss function, a mean absolute error lossfunction, a Huber loss function, or a weighted sum of squareddifferences loss function, and/or the like.

In some embodiments, the first patch may include, without limitation,one of a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like,where the first set of color or brightfield images may include, but isnot limited to, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. In some cases, the second image may include, withoutlimitation, one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like, where the second set of color or brightfield images mayinclude, but is not limited to, a second R image, a second G image, anda second B image, and/or the like. The second fluorescence image mayinclude, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike. In some instances, the predicted virtually stained image patch mayinclude, without limitation, one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, where the third set of color orbrightfield images may include, but is not limited to, a third R image,a third G image, and a third B image, and/or the like. The thirdfluorescence image may include, but is not limited to, at least one of athird autofluorescence image or a third labelled fluorescence imagehaving one or more channels with different excitation or emissioncharacteristics, and/or the like. The third spectral image may include,but is not limited to, at least one of a third Raman spectroscopy image,a third NIR spectroscopy image, a third multispectral image, a thirdhyperspectral image, or a third full spectral image, and/or the like.

According to some embodiments, operating on the encoded first patch togenerate the color vector may comprise operating on the encoded firstpatch to generate a color vector, using a color vector output and one ofa model architecture, a statistical model-based system, or adeterministic analysis system, and/or the like. In some instances, themodel architecture may comprise at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”), and/or the like. In some cases, the color vector may be one ofa fixed color vector or a learned color vector that is not based on theencoded first patch, and/or the like.

In some embodiments, the image of the virtual stain may include, but isnot limited to, one of a 3-channel image of the virtual stain, aRGB-transform image of the virtual stain, or a logarithmic transformimage of the virtual stain, and/or the like. In some instances,generating the predicted virtually stained image patch may include,without limitation, adding the generated image of the virtual stain tothe received first patch to produce the predicted virtually stainedimage patch. In some cases, determining the first loss value between thepredicted virtually stained image patch and the second patch may beperformed without adding the generated image of the virtual stain to thereceived first patch.

According to some embodiments, the first loss value may include, withoutlimitation, one of a pixel loss value between each pixel in thepredicted virtually stained image patch and a corresponding pixel in thesecond patch or a generative adversarial network (“GAN”) loss valuebetween the predicted virtually stained image patch and the secondpatch, and/or the like. In some instances, the GAN loss value may begenerated based on one of a minimax GAN loss function, a non-saturatingGAN loss function, a least squares GAN loss function, or a WassersteinGAN loss function, and/or the like.

Alternatively, training the AI system to update Model F may comprise:receiving, with the AI system, the first patch; receiving, with the AIsystem, the second patch; generating, with a second model of the AIsystem, an image of the virtual stain; adding the generated image of thevirtual stain to the received first patch to produce a predictedvirtually stained image patch; determining a first loss value betweenthe predicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value.

In some embodiments, the second model may include, without limitation,at least one of a convolutional neural network (“CNN”), a U-Net, anartificial neural network (“ANN”), a residual neural network (“ResNet”),an encode/decode CNN, an encode/decode U-Net, an encode/decode ANN, oran encode/decode ResNet, and/or the like.

According to some embodiments, the computing system may receive a fourthimage, the fourth image comprising one of a fourth FOV of the firstbiological sample different from the first FOV and the second FOV or afifth FOV of a second biological sample, where the second biologicalsample may be different from the first biological sample; and mayidentify, using Model G*, second instances of features of interest inthe second biological sample, based at least in part on the fourth imageand based at least in part on training of Model G* using at least thethird patch comprising the virtual stain of the first aligned imagepatches. According to some embodiments, the computing system maygenerate, using Model G*, a clinical score, based at least in part onthe identified second instances of features of interest.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample (including, but notlimited to, at least one of first antigens, first nuclei, first cellwalls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. In some cases, the third patchmay comprise one of highlighting of the first features of interest bythe first stain and highlighting of second features of interest in thefirst biological sample (including, but not limited to, at least one ofsecond antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the virtual stain that simulates the secondstain having been applied to the first biological sample or highlightingof the first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest. In someinstances, the second stain may be one of the same as the first stainbut used to stain the second features of interest or different from thefirst stain.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image or a firstlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. In someinstances, the second image may include, without limitation, one of asecond set of color or brightfield images, a second fluorescence image,a second phase image, or a second spectral image, and/or the like, wherethe second set of color or brightfield images may include, but is notlimited to, a second R image, a second G image, and a second B image,and/or the like. The second fluorescence image may include, but is notlimited to, at least one of a second autofluorescence image or a secondlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The secondspectral image may include, but is not limited to, at least one of asecond Raman spectroscopy image, a second NIR spectroscopy image, asecond multispectral image, a second hyperspectral image, or a secondfull spectral image, and/or the like. In some cases, the third patch mayinclude, without limitation, one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, where the third set of color orbrightfield images may include, but is not limited to, a third R image,a third G image, and a third B image, and/or the like. The thirdfluorescence image may include, but is not limited to, at least one of athird autofluorescence image or a third labelled fluorescence imagehaving one or more channels with different excitation or emissioncharacteristics, and/or the like. The third spectral image may include,but is not limited to, at least one of a third Raman spectroscopy image,a third NIR spectroscopy image, a third multispectral image, a thirdhyperspectral image, or a third full spectral image, and/or the like.

In some embodiments, training the AI system to update Model F maycomprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may include,but is not limited to, one of a mean squared error loss function, a meansquared logarithmic error loss function, a mean absolute error lossfunction, a Huber loss function, or a weighted sum of squareddifferences loss function, and/or the like.

According to some embodiments, the first loss value may include, withoutlimitation, one of a pixel loss value between each pixel in thepredicted virtually stained image patch and a corresponding pixel in thesecond patch or a generative adversarial network (“GAN”) loss valuebetween the predicted virtually stained image patch and the secondpatch, and/or the like. In some cases, the GAN loss value may begenerated based on one of a minimax GAN loss function, a non-saturatingGAN loss function, a least squares GAN loss function, or a WassersteinGAN loss function, and/or the like.

In yet another aspect, the computing system may receive a first image ofa first biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample; and may identify, using afirst model (“Model G*”) that is generated or updated by a trained AIsystem, first instances of features of interest in the first biologicalsample, based at least in part on the first image and based at least inpart on training of Model G* using at least a first patch comprising avirtual stain of first aligned image patches, the first patch beinggenerated by a second AI model (Model F) that is generated or updated bythe trained AI system by using a second patch. In some cases, the firstaligned image patches may comprise an extracted portion of first alignedimages. The first aligned images may comprise a second image and a thirdimage that have been aligned. The second image may comprise a second FOVof a second biological sample that is different from the firstbiological sample that has been stained with a first stain. The secondpatch may comprise the extracted portion of the second image. The thirdimage may comprise a third FOV of the second biological sample that hasbeen stained with at least a second stain.

In general, in some embodiments, the training of the (AI) Models mayleverage the following: (1) A set of biological samples (i.e., one ormore samples) represents the tissue of interest; (2) Each such samplemay also have annotation associated with it (e.g., nuclei position,pathologist marking, other labeling as described herein, etc.); (3) Eachsuch biological sample may be stained and images may be scanned orcaptured in the following manner: (a) “First” staining (for exampleH+DAB, although not limited to such), followed by at least one multiple“sequential” staining steps; (b) After each staining step, the tissue isscanned, resulting in production of a set of Whole-Slide-Image (“WSI”)for each biological sample produced, one WSI for each staining; (4) Foreach biological sample, a large set of regions (not just one) may beselected, such a region in some cases being referred to as a FOV orsample FOV; (5) For each such FOV (or sample FOV), a “patches-set” mayreflect the collection of corresponding image patches (one for each WSIfor this biological sample), together with the “annotation patch,”corresponding to the associated annotation for the biological sample.According to some embodiments, the scans of the images of one or morebiological samples may include, without limitation, at least one of abrightfield scan, a fluorescence scan (e.g., autofluorescence scan, orthe like), and/or a spectral scan, and/or the like, with the patches-setincluding such multitude of scans (if available for that biologicalsample). Here, a patches-set may correspond to a specific FOV(s), withina specific biological sample, and may have one member for eachstaining + annotation. The Training process takes this whole multitudeof patches-sets and trains an (AI) model with them. This model will beable to predict various annotations and/or to generate a clinical score,given an arbitrary image patch from a scan of a biological samplestained with only the “First” staining above. Training of the (AI)Models is not necessarily limited to training based on only onebiological sample, or based on there being only one FOV, or based onthere being only one sequential staining. Rather, training of the (AI)Models may be based on training of one or more biological samples, basedon one or more FOVs of each biological sample, and/or based on a singlestaining or sequential staining or unmixing of particular staining,and/or the like. Herein also, “the (AI) Models” may refer to AI modelsor to models that are not necessarily AI-based (e.g., statisticalmodel-based, deterministic analysis, etc.).

These and other functions of the system 100 (and its components) aredescribed in greater detail below with respect to FIGS. 4-9 .

FIGS. 4A-4C (collectively, “FIG. 4 ”) are process flow diagramsillustrating various non-limiting examples 400 and 400′of training ofone or more artificial intelligence (“AI”) models and non-limitingexample 400″of associated inferencing performed by a trained AI model,when implementing sequential imaging of biological samples forgenerating training data for developing deep learning based models forimage analysis, for cell classification, and/or for features of interestidentification of biological samples, in accordance with variousembodiments. FIG. 4 is directed to the process of generating groundtruth for developing accurate AI models for biological imageinterpretation, based at least in part on images of a first biologicalsample depicting sequential staining of the first biological sample.Such a process, in some cases, may be performed by computing system 305a or 305 b and/or AI system 305 d or 305 c (collectively, “computingsystem”) of FIG. 3 , or the like.

Although FIG. 4 is focused on specific examples that utilize particularcomponents (including, but not limited to, whole slide images (“WSIs”),nuclei detection, using pairs of patches or images, using alignment ofimages, etc.) in a particular order, the various embodiments are not solimited, and some of these components may be replaced with other typesof components, may be omitted, may be reordered, or may otherwise bychanged, as necessary or as desired, without deviating from the scope ofthe various embodiments. For instance, images of biological samplesother than WSIs may be used, detection of other features of interest(not necessarily nuclei or cells) may be utilized, sets of three or morepatches or images may be used rather than pairs, and/or images mayalready have been aligned or do not require alignment, or the like.Further, although specific examples of biological markers or stains maybe referred to with respect to FIG. 4 , this is merely for purposes ofillustration, and the various embodiments are not so limited, as anysuitable biological marker(s) and/or stain(s) may be used, asappropriate or as desired.

With reference to the non-limiting embodiment 400 of FIG. 4A, a firstimage 405 of a first biological sample and a second image 410 of thefirst biological sample may be received. According to some embodiments,the first biological sample may include, without limitation, one of ahuman tissue sample, an animal tissue sample, a plant tissue sample, oran artificially produced tissue sample, and/or the like. The first image405 (which may comprise a WSI, although not limited to such) may includea first field of view (“FOV”) of the first biological sample that hasbeen stained with a first stain. The second image 410 (which maycomprise a WSI, although not limited to such) may include a second FOVof the first biological sample that has been stained with a secondstain. In some cases, while the first image 405 may include highlightingof first features of interest in the first biological sample (including,but not limited to, at least one of first antigens, first nuclei, firstcell walls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain, the second image410 may include either highlighting of second features of interest inthe first biological sample (including, but not limited to, at least oneof second antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the second stain in addition to highlightingof the first features of interest by the first stain or highlighting ofthe second features of interest by the second stain without highlightingof the first features of interest by the first stain. In some instances,the second features of interest may be different from the first featuresof interest.

In some embodiments, the second stain either may be the same as thefirst stain but used to stain the second features of interest or may bedifferent from the first stain. In some instances, the first stain andthe second stain may each include, but is not limited to, at least oneof Hematoxylin, Acridine orange, Bismarck brown, Carmine, Coomassieblue, Cresyl violet, Crystal violet, 4′,6-diamidino-2-phenylindole(“DAPI”), Eosin, Ethidium bromide intercalates, Acid fuchsine, Hoechststain, Iodine, Malachite green, Methyl green, Methylene blue, Neutralred, Nile blue, Nile red, Osmium tetroxide, Propidium Iodide, Rhodamine,Safranine, programmed death-ligand 1 (“PD-L1”) stain,3,3′-Diaminobenzidine (“DAB”) chromogen, Magenta chromogen, cyaninechromogen, cluster of differentiation (“CD”) 3 stain, CD20 stain, CD68stain, 40S ribosomal protein SA (“p40”) stain, antibody-based stain, orlabel-free imaging marker, and/or the like. In some cases, the firstimage 405 and the second image 410 may each include, without limitation,one of a set of color or brightfield images, a fluorescence image, or aspectral image, and/or the like, where the set of color or brightfieldimages may include, but is not limited to, a red-filtered (“R”) image, agreen-filtered (“G”) image, and a blue-filtered (“B”) image, and/or thelike.

As depicted in FIG. 4A (with the notation “WSI” being at a differentangle in the second image 410 compared with corresponding notation inthe first image 405), the first image 405 and the second image 410 mayinitially be misaligned, and thus may require alignment by alignmentsystem 415. The alignment system 415 may perform alignment of the firstimage 405 relative to the second image 410 (or vice versa) either usingmanual input from a user (e.g., pathologist or other operator, or thelike) or via autonomous alignment processes (in some cases, utilizing AIsystem functionality, or the like). In some cases, autonomous alignmentmay include, but is not limited to, alignment using at least one ofautomated global alignment techniques or optical flow alignmenttechniques, or the like.

In some embodiments, automated global alignment may include thefollowing computational details: (i) finding foreground tiles (in somecases, using entropy threshold criterion, or the like); (ii) performingfeature-based (e.g., with oriented fast and rotated brief (“ORB”)features, or the like) matching on low resolution images (e.g., defaultpyramid level 4, or the like); and (iii) computing affine alignmenttransformation from matched features. Computational steps (i) to (iii)are applicable to both serial and sequential staining. For sequentialstaining, the following additional details may apply for automatedglobal alignment: (iv) finding all high resolution (e.g., level 0) tilesspanning the matched low resolution features; (v) matching highresolution tiles based on low resolution alignment; (vi) performingfeature based matching on high resolution features within matched tiles;and (vii) computing global affine alignment as composition of the highand low resolution alignment transformation matrices; and/or the like.

Even with sequential staining, there may be misalignments anddeformations between scans. Global, affine alignment accounts fortranslation, rotation, or uniform deformation of the whole slide image,but cannot account for non-uniform deformation of the slide image. Largescale deformation may result in misalignment of paired FOVs, while smallscale deformation may result in non-uniform misalignment within a FOV(which may be apparent as horizontal stretching of the FOV, forexample). Such misalignment introduces noise to the training of modelsbased on the sequentially stained image as ground truth. Alignmentcharacterization may be optimized as follows: (a) optimization may beperformed on either H channel alone, or on the sum of H+DAB channels(combined); (b) optimization may be performed by non-overlapping squaremicro-patches of equal size having patch sides in 1000, 500, 100, or 50pixels in length; (c) manual global rigid alignment may be performed formicro-patch matching; (d) for each patch-pair, rigid transform (i.e.,rotation + translation) parameters may be found that minimize themis-match cost; (e) no continuity constraints across patches may beenforced - free optimization; (f) several cost functions may beevaluated by patch-level random self-alignment test, with the best (moststable) results being attained by pixel-wise MSE (L2) loss or the like;and (g) each micro-patch may be padded by adding extra pixels around thepatch to allow for required transformation range, which was not alwayspossible (due to FOV edges).

Quantitative evaluation of alignment characterization may alternativelyor additionally be performed as follows: (1) quantitative evaluation maybe performed by measuring cell-center distances in paired FOVS, withFOVs annotated using the predictions of segmentation models, withsegmentation net trained or retrained and running on the H channel only,and with centroids calculated using region_props from segmentationpredictions; (2) annotation in paired FOVs may be matched by k=1 nearestneighbour (“NN”) matching; (3) NN matching may be applied with thresholdon NN distance ratio (k=1, 2); (4) NN matching may also be applied withthreshold on a global distance threshold; and (5) matched distancehistograms pre and post-optimization may be generated for each FOV.Distance histograms pre and post optimization for micro-patches may beused for quantitative evaluation of the alignment characterization.

According to some embodiments, optical flow alignment may utilize denseinverse search (“DIS”), which may include the following characteristics:(A) optical flow dense (per-pixel) displacement field; (B) “coarse tofine” micro-patch based optimization with partial patch overlap; (C)utilizing an open-sourced implementation (e.g., Python bindings to a Cimplementation, or the like); (D) integrating into a computing platform(e.g., Matlab calling Python, or the like); (E) the algorithm’s patchsize and overlap may be empirically tuned for robustness over varioustissue samples and various image pyramid levels; and (F) achievinglatency per tile (including padding) of < 100 ms (in some evaluations,~40-50 ms was achieved). The algorithm details for optical flowalignment may include the following: (I) dividing the image into a gridof equally sized micro-patches; (II) for each micro-patch in the image,finding the displacement to its optimal position in the paired image(under MSE loss, or the like); (III) starting with coarse (lowresolution) patches, and using optimal displacement as the initialcondition for the next (finer) level; (IV) overlapping partialmicro-patch with similarity-weighted densification for continuity andsmoothness; and (V) applying variational refinement on intensityfunctions for further smoothness and conservation constraints (e.g.,divergence of brightness, color, and gradients, or the like). Distancehistograms pre and post optimization for DIS may be used forquantitative evaluation of the optical flow alignment. In summary, toaccount for local, non-uniform deformations, a second tier of localalignment -namely, optical flow alignment - may be used. Optical flowalignment refines the global alignment. Optical flow alignment canaccount for paired-FOV misalignment as well as non-uniform misalignmentwithin a FOV. With optical flow alignment, pixel-perfect registration ofthe sequentially stained images can be achieved.

In some cases, the received first and second images may be pre-alignedprior to being received, or may not require alignment (such as in thecase of output images of a fluorescence staining scanning protocol, orthe like), and thus alignment by the alignment system 415 may beomitted. After alignment (if needed), aligned image 420 may be used toextract at least one second image patch 425 (a plurality of which isdepicted in FIG. 4A). In some instances, aligned image 420 may bedivided into equal sized (and shaped) grids, with each image patch 425representing a grid. In some cases, the grids may be of different sizesand/or different shapes (e.g., rectangles, squares, trapezoids, circles,triangles, or other polygons, etc.).

Meanwhile, a nuclei detection system 430 (or other feature detectionsystem (not shown), or the like) may be used to detect nuclei (or otherfeatures of interest) in the first image 405, with the resultant image435 being used to extract at least one first image patch 440 (aplurality of which is depicted in FIG. 4A), in a similar manner as theat least one second image patch 425, each of the at least one firstimage patch 440 corresponding to each of the at least one second imagepatch 425 to form patch pairs (although a set of corresponding three ormore images, not limited to a pair, may be used, such as in the case ofa third image (not shown) being used to highlight a third stain or acombination of stains).

The at least one first image patch 440 and the at least one second imagepatch 425 (or patch pairs, or the like) may serve as input images fortraining a first cell classifier net, a first AI module, or a firstmodel (“G*”; collectively, “first AI model G*” or “first (AI) model G*”or “Model G*” or the like) 445, which may also be referred to herein asa “ground truth generator” or the like. The first model G* 445 (whichmay, in some cases, receive external labelling data, or the like, inaddition to the nuclei or other feature detection data, or the like) maygenerate first instance classification of features of interest (“GroundTruth”) in the first biological sample, based at least in part on thealigned image patches 440 and 425, based at least in part on thelabeling of instances of features of interest contained in the at leastone second patch 425, and (in some cases) based at least in part onadditional external labelling data.

The generated first instance classification or Ground Truth, along withthe image 435 (containing results of the nuclei or other featuredetection, or the like) may be used as inputs to train a second cellclassifier net, a second AI module, or a second AI model (“G”;collectively, “second AI model G” or the like) 450 to identify instancesof features of interest in the first biological sample. The process flowas depicted in FIG. 4A represents a computational pipeline (alsoreferred to herein as “pipeline A”) for generating ground truth based onimages of a sequentially stained biological sample, and using suchgenerated ground truth to train another AI model (in this case, model G)to identify features of interest in biological samples. Such groundtruth generation would improve upon the technical field of annotationcollection or autonomous annotation collection, by increasing the speedof annotation collection, while, in some cases, identifying features ofinterest that are difficult if not impossible for a human to identify.

Turning to the non-limiting embodiment 400′of FIG. 4B, an alreadytrained first model G* 445′(which may be trained in a manner similar tothat described above with respect to FIG. 4A, or the like) may be usedto generate ground truth to train the second AI model G 450, which mayalso use, as inputs for training, image 435′(containing results of thenuclei or other feature detection, or the like, by nuclei detectionsystem (or other feature detection system) 430′being used to detectnuclei or other features in first image 405′). The embodiment 400′mayotherwise be similar to embodiment 400 of FIG. 4A.

Referring to the non-limiting embodiment 400″of FIG. 4C, after thesecond AI model G has been trained, the second AI model G 450′may beused to receive as an input, input image 455, and to identify featuresof interest in input image 455 and to generate an output 460 containinglabelling of identified features of interest in input image 455. In someembodiments, the output 460 may include, without limitation, at leastone of a clinical score and/or a labelled image (identifying features ofinterest), or the like. Such inferencing or overall scoring or diagnosiswould improve upon the technical field of annotation collection orautonomous annotation collection, by improving precision and accuracy inidentifying features of interest and/or avoiding observer/pathologistbiases or human error effects, or the like. Additionally, pipeline Aprovides a simple, straight forward approach, with relatively lowcomplexity (in some cases, using binary classification, or the like).

These and other features are similar, if not identical to the features,processes, and techniques described herein with respect to FIGS. 3 and5-9 , or the like.

FIGS. 5A-5N (collectively, “FIG. 5 ”) are schematic diagramsillustrating a non-limiting example 500 of various process stepsperformed by the system when implementing sequential imaging ofbiological samples for generating training data for developing deeplearning based models for image analysis, for cell classification,and/or for features of interest identification of biological samples inconjunction with corresponding images of the biological sample duringeach process (and in some cases, at varying levels of magnification), inaccordance with various embodiments.

Although FIG. 5 is focused on specific examples that utilize particularcomponents (including, but not limited to, whole slide images (“WSIs”),nuclei segmentation, using hematoxylin (“H”) and the combination of Hand Magenta (e.g., attached/staining p40 antibody, or the like) (“H+M”),generating p40 ground truth, using alignment of images, etc.) in aparticular order, the various embodiments are not so limited, and someof these components may be replaced with other types of components, maybe omitted, may be reordered, or may otherwise by changed, as necessaryor as desired, without deviating from the scope of the variousembodiments. For instance, images of biological samples other than WSIsmay be used, detection of other features of interest (not necessarilynuclei or cells) may be utilized, stains other than H or H+M may beused, ground truth other than for p40 antibody based ground truth may begenerated, and/or images may already have been aligned or do not requirealignment, or the like.

The non-limiting embodiment 500 of FIG. 5 demonstrates automaticprocessing of p40-positive (or p40+) tumor nuclear marker in squamouscell carcinoma sequential stain tissue slides for ground truthgeneration (i.e., by cell classifier G* without external labeling inpipeline A as shown in FIG. 4A). This ground truth is then used fortraining nuclei patch based classification deep neural network (i.e.,cell classifier G in pipeline A as shown in FIG. 4A) on Hematoxylinpatches extracted from WSI around detected nuclei. In some instances,classifier architecture known as Resnet50 or the like may be used. Thecell classifier (or neural network) may be trained from scratch on oneor more 101×101×RGB Hematoxylin patches extracted from one or more WSIs.Performances of the trained classifier, evaluated on validation FOVs(similar to that depicted in FIG. 8E, or the like), that were not usedduring training, from the same WSIs, can be seen in the confusion matrixshown in FIG. 5N.

With reference to FIGS. 5A-5M, an alternative depiction of computationalpipeline A for WSI ground truth generator from sequential stains fortraining nuclei patch based classification deep neural network is shownin (a). A Hematoxylin patch is shown in (b) as well as in FIG. 5B, whilea Hematoxylin patch overlaid with segmented nuclei is shown in (c) aswell as in FIG. 5C, and an aligned Hematoxylin+p40 Magenta marker patchoverlaid with segmented nuclei from Hematoxylin patch is shown in (d) aswell as in FIG. 5D or FIG. 5E. The patch as depicted in (d) afterunmixing following white balancing is shown in (e) as well as in FIG. 5For FIG. 5G. The patch as depicted in (e) after binarization, integrationover segmented nuclei, thresholding, and grouping calculated nucleicentroids to p40+ (pink colored points) and p40- (cyan colored points)is shown in (f) as well as in FIG. 5H. The resultant patch depictingnuclei shown as p40+ (pink colored points) and p40- (cyan coloredpoints) is shown in (g) as well as in FIG. 51 . The patches shown in(a)-(g) as well as in FIGS. 5B-5I are depicted in 40X magnification. 10Xmagnification Hematoxylin region marked in p40+/p40-points is shown in(h). FIG. 5J illustrates the patch as depicted FIG. 5I in 20Xmagnification, while FIGS. 5K, 5L, and 5M illustrate the patch asdepicted FIG. 5I in 10X, 5X, and 1X magnification, respectively. Asshown in the non-limiting embodiment of FIG. 5M, the whole slide imagemay contain 420,016 nuclei, 95,751 p40+, and 324,265 p40-.

FIG. 5N depicts a non-limiting example of evaluation of trained nucleipatch based classification deep neural network on validation FOVs from11 WSIs, embodied by a confusion matrix. The output and target representthe predicted and ground truth classes, respectively. Precision andrecall are represented by blue colored numbers. False negative rate(“FNR”) and false positive rate (“FPR”) are represented by red colorednumbers. The cell in the bottom right of the confusion matrix depictsthe overall accuracy in blue colored numbers. For the classificationevaluated in the accuracy, precision, recall, confusion matrix of FIG.5N, the Resnet50 was trained from scratch using about 3 million nucleifrom the 11 WSIs, with evaluation of 0.3 million validation nuclei from11 WSIs.

FIGS. 6A-6E (collectively, “FIG. 6 ”) are flow diagrams illustrating amethod 600 for implementing sequential imaging of biological samples forgenerating training data for developing deep learning based models forimage analysis, for cell classification, and/or for features of interestidentification of biological samples, in accordance with variousembodiments. Method 600 of FIG. 6A continues onto FIG. 6B following thecircular marker denoted, “A,” and returns to FIG. 6A following thecircular marker denoted, “B.” Method 600 of FIG. 6A continues onto FIG.6C following the circular marker denoted, “C.”

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 600 illustrated byFIG. 6 can be implemented by or with (and, in some cases, are describedbelow with respect to) the systems, examples, or embodiments 300, 400,400′, 400″, and 500 of FIGS. 3, 4A, 4B, 4C, and 5 , respectively (orcomponents thereof), such methods may also be implemented using anysuitable hardware (or software) implementation. Similarly, while each ofthe systems, examples, or embodiments 300, 400, 400′, 400″, and 500 ofFIGS. 3, 4A, 4B, 4C, and 5 , respectively (or components thereof), canoperate according to the method 600 illustrated by FIG. 6 (e.g., byexecuting instructions embodied on a computer readable medium), thesystems, examples, or embodiments 300, 400, 400′, 400″, and 500 of FIGS.3, 4A, 4B, 4C, and 5 can each also operate according to other modes ofoperation and/or perform other suitable procedures.

In the non-limiting embodiment of FIG. 6A, method 600, at block 605, maycomprise receiving, with a computing system, a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample that has been stained with afirst stain. Method 600 may comprise receiving, with the computingsystem, a second image of the first biological sample, the second imagecomprising a second FOV of the first biological sample that has beenstained with a second stain (block 610).

In some embodiments, the computing system may include, withoutlimitation, one of a computing system disposed in a work environment, aremote computing system disposed external to the work environment andaccessible over a network, a web server, a web browser, or a cloudcomputing system, and/or the like. In some instances, the workenvironment may include, but is not limited to, at least one of alaboratory, a clinic, a medical facility, a research facility, ahealthcare facility, or a room, and/or the like. In some cases, thefirst biological sample may include, without limitation, one of a humantissue sample, an animal tissue sample, a plant tissue sample, or anartificially produced tissue sample, and/or the like, and the featuresof interest may include, but are not limited to, at least one of normalcells, abnormal cells, damaged cells, cancer cells, tumors, subcellularstructures, or organ structures, and/or the like.

Merely by way of example, in some cases, the first image may comprisehighlighting of first features of interest in the first biologicalsample (including, but not limited to, at least one of first antigens,first nuclei, first cell walls, first cell structures, first antibodies,first normal cells, first abnormal cells, first damaged cells, firstcancer cells, first tumors, first subcellular structures, first organstructures, or other features of interest, or the like) by the firststain that had been applied to the first biological sample. The secondimage may comprise one of highlighting of second features of interest inthe first biological sample (including, but not limited to, at least oneof second antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the second stain that had been applied to thefirst biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain, the second features ofinterest being different from the first features of interest. In someinstances, the second stain may be one of the same as the first stainbut used to stain the second features of interest or different from thefirst stain.

According to some embodiments, the first stain and the second stain mayeach include, without limitation, at least one of Hematoxylin, Acridineorange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystalviolet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.

In some embodiments, the first image may include, but is not limited to,one of a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like.The first set of color or brightfield images may include, withoutlimitation, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. Similarly, the second image may include, but is notlimited to, one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like. The second set of color or brightfield images mayinclude, without limitation, a second R image, a second G image, and asecond B image, and/or the like. The second fluorescence image mayinclude, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike.

At optional block 615, method 600 may comprise aligning, with thecomputing system, the first image with the second image to create firstset of aligned images, by aligning one or more features of interest inthe first biological sample as depicted in the first image with the sameone or more features of interest in the first biological sample asdepicted in the second image. In some instances, aligning the firstimage with the second image may either be performed manually usingmanual inputs to the computing system or performed autonomously, whereautonomous alignment may comprise alignment using at least one ofautomated global alignment techniques or optical flow alignmenttechniques, or the like.

Method 600 may continue onto the process at block 620. Alternatively,method 600 may continue onto the process at block 635 in FIG. 6Bfollowing the circular marker denoted, “A,” and may return from FIG. 6Bto the process at block 620 in FIG. 6A following the circular markerdenoted, “B.”

At block 635 in FIG. 6B (following the circular marker denoted, “A”),method 600 may comprise receiving, with the computing system, a thirdimage of the first biological sample, the third image comprising a thirdFOV of the first biological sample that has been stained with a thirdstain different from each of the first stain and the second stain.Method 600 may further comprise, at block 640, autonomously performing,with the computing system, one of: aligning the third image with each ofthe first image and the second image to create second aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the third image with the same one or more features ofinterest in the first biological sample as depicted in each of the firstimage and the second image (block 640 a); or aligning the third imagewith the first aligned images to create second aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the third image with the same one or more features ofinterest in the first biological sample as depicted in the first alignedimages (block 640 b). Method 400 may return to the process at block 620in FIG. 6A following the circular marker denoted, “B.”

Method 600 may further comprise, at block 620, autonomously creating,with the computing system, a first set of image patches based on thefirst image and the second image (and, in some cases, the third image),by extracting a portion of the first image and by extracting acorresponding portion of the second image (and, in some cases, byextracting a corresponding portion of the third image). In some cases,the first set of image patches may include, without limitation, a firstpatch corresponding to the extracted portion of the first image and asecond patch corresponding to the extracted portion of the second image(and, in some cases, a third patch corresponding to the extractedportion of the third image). In some instances, the first set of imagepatches may comprise labeling of instances of features of interest inthe first biological sample that is based at least in part on at leastone of information contained in the first patch, information containedin the second patch, information contained in one or more externallabeling sources, and/or information contained in the third patch (ifapplicable).

In some cases, the third image may comprise one of highlighting of thirdfeatures of interest in the first biological sample (including, but notlimited to, at least one of third antigens, third nuclei, third cellwalls, third cell structures, third antibodies, third normal cells,third abnormal cells, third damaged cells, third cancer cells, thirdtumors, third subcellular structures, third organ structures, or otherfeatures of interest, or the like) by the third stain that had beenapplied to the first biological sample in addition to highlighting ofthe first features of interest by the first stain and highlighting ofthe second features of interest by the second stain, highlighting of thethird features of interest by the third stain in addition to only one ofhighlighting of the first features of interest by the first stain orhighlighting of the second features of interest by the second stain, orhighlighting of the third features of interest by the third stainwithout any of highlighting of the first features of interest by thefirst stain or highlighting of the second features of interest by thesecond stain, the third features of interest being different from eachof the first features of interest and the second features of interest.

In some instances, the third image may include, but is not limited to,one of a third set of color or brightfield images, a third fluorescenceimage, a third phase image, or a third spectral image, and/or the like.The third set of color or brightfield images may include, withoutlimitation, a third R image, a third G image, and a third B image,and/or the like. The third fluorescence image may include, but is notlimited to, at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may include, but is not limited to, at least one of athird Raman spectroscopy image, a third NIR spectroscopy image, a thirdmultispectral image, a third hyperspectral image, or a third fullspectral image, and/or the like.

Method 600 may further comprise training, using an artificialintelligence (“AI”) system, a first model (“Model G*”) to generate firstinstance classification of features of interest (“Ground Truth”) in thefirst biological sample, based at least in part on the first set ofimage patches and the labeling of instances of features of interestcontained in the first set of image patches (block 625); and training,using the AI system, a second AI model (“Model G”) to identify instancesof features of interest in the first biological sample, based at leastin part on the first patch and the first instance classification offeatures of interest generated by Model G* (block 630). In someembodiments, the AI system may include, but is not limited to, at leastone of a machine learning system, a deep learning system, a modelarchitecture, a statistical model-based system, or a deterministicanalysis system, and/or the like. In some instances, the modelarchitecture may comprise at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”), and/or the like.

Method 600 may continue onto the process at block 645 in FIG. 6Cfollowing the circular marker denoted, “C.”

At block 645 in FIG. 6C (following the circular marker denoted, “C”),method 600 may comprise receiving, with the computing system, a fourthimage, the fourth image comprising one of a fourth FOV of the firstbiological sample different from the first FOV and the second FOV or afifth FOV of a second biological sample, where the second biologicalsample is different from the first biological sample. Method 600 mayfurther comprise, at block 650, identifying, using Model G, secondinstances of features of interest in the second biological sample, basedat least in part on the fourth image and based at least in part ontraining of Model G using the first patch and the first instanceclassification of features of interest generated by Model G*. In somecases, the fourth image may comprise the fifth FOV and may furthercomprise only highlighting of first features of interest in the secondbiological sample by the first stain that had been applied to the secondbiological sample. According to some embodiments, method 600 may furthercomprise generating, using Model G, a clinical score, based at least inpart on the identified second instances of features of interest.

With reference to FIG. 6D, method 600 may comprise receiving, with acomputing system, first instance classification of features of interest(“Ground Truth”) in a first biological sample that has been sequentiallystained, the Ground Truth having been generated by a trained first model(“Model G*”) that has been trained or updated by an artificialintelligence (“AI”) system (block 655); and training, using the AIsystem, a second AI model (“Model G”) to identify instances of featuresof interest in the first biological sample, based at least in part onthe first instance classification of features of interest generated byModel G* (block 660). In some cases, the Ground Truth may be generatedby using first aligned image patches and labeling of instances offeatures of interest contained in the first aligned image patches,wherein the first aligned image patches comprise an extracted portion offirst aligned images, wherein the first aligned images comprise a firstimage and a second image that have been aligned, wherein the first imagecomprises a first FOV of the first biological sample that has beenstained with a first stain, wherein the second image comprises a secondFOV of the first biological sample that has been stained with a secondstain, wherein the first aligned image patches comprise labeling ofinstances of features of interest as shown in the extracted portion ofthe first aligned images.

Referring to FIG. 6E, method 600 may comprise, at block 665, receiving,with a computing system, a first image of a first biological sample, thefirst image comprising a first field of view (“FOV”) of the firstbiological sample. At block 670, method 600 may comprise identifying,using a first artificial intelligence (“AI”) model (“Model G”) that istrained by an AI system, first instances of features of interest in thefirst biological sample, based at least in part on the first image andbased at least in part on training of Model G using a first patch andfirst instance classification of features of interest generated by asecond model (Model G*) that is trained by the AI system by using firstaligned image patches and labeling of instances of features of interestcontained in the first patch. In some instances, the first aligned imagepatches may comprise an extracted portion of first aligned images. Thefirst aligned images may comprise a second image and a third image thathave been aligned. The second image may comprise a second FOV of asecond biological sample that has been stained with a first stain, thesecond biological sample being different from the first biologicalsample. The third image may comprise a third FOV of the secondbiological sample that has been stained with a second stain. The secondpatch may comprise labeling of instances of features of interest asshown in the extracted portion of the second image of the first alignedimages. The first image may comprise highlighting of first features ofinterest in the second biological sample by the first stain that hadbeen applied to the second biological sample. The second patch maycomprise one of highlighting of second features of interest in thesecond biological sample by the second stain that had been applied tothe second biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain.

FIGS. 7A-7D (collectively, “FIG. 7 ”) are process flow diagramsillustrating various non-limiting examples 700, 700″, and 700‴ oftraining of an artificial intelligence (“AI”) model to generate avirtual stain and a non-limiting example 700′ of inferencing performedby a trained AI model when implementing sequential imaging of biologicalsamples for generating training data for developing deep learning basedmodels for virtual staining of biological samples and for generatingtraining data for developing deep learning based models for imageanalysis, cell classification, and/or features of interestidentification of biological samples, in accordance with variousembodiments. FIG. 7 is directed to the process of generating virtualstaining for simulating sequential staining of biological samples fordeveloping accurate AI models for biological image interpretation. Sucha process, in some cases, may be performed by computing system 305 a or305 b and/or AI system 305 d or 305 c (collectively, “computing system”)of FIG. 3 , or the like.

Although FIG. 7 is focused on specific examples that utilize particularcomponents (including, but not limited to, whole slide images (“WSIs”),nuclei detection, using pairs of patches or images, using alignment ofimages, etc.) in a particular order, the various embodiments are not solimited, and some of these components may be replaced with other typesof components, may be omitted, may be reordered, or may otherwise bychanged, as necessary or as desired, without deviating from the scope ofthe various embodiments. For instance, images of biological samplesother than WSIs may be used, detection of other features of interest(not necessarily nuclei or cells) may be utilized, sets of three or morepatches or images may be used rather than pairs, and/or images mayalready have been aligned or do not require alignment, or the like.Further, although specific examples of biological markers or stains maybe referred to with respect to FIG. 7 , this is merely for purposes ofillustration, and the various embodiments are not so limited, as anysuitable biological marker(s) and/or stain(s) may be used, asappropriate or as desired.

With reference to the non-limiting embodiment 700 of FIG. 7A, a firstimage 705 of a first biological sample and a second image 710 of thefirst biological sample may be received. According to some embodiments,the first biological sample may include, without limitation, one of ahuman tissue sample, an animal tissue sample, a plant tissue sample, oran artificially produced tissue sample, and/or the like. The first image705 (which may comprise a WSI, although not limited to such) may includea first field of view (“FOV”) of the first biological sample that hasbeen stained with a first stain. The second image 710 (which maycomprise a WSI, although not limited to such) may include a second FOVof the first biological sample that has been stained with a secondstain. In some cases, while the first image 705 may include highlightingof first features of interest in the first biological sample (including,but not limited to, at least one of first antigens, first nuclei, firstcell walls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain, the second image710 may include either highlighting of second features of interest inthe first biological sample (including, but not limited to, at least oneof second antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the second stain in addition to highlightingof the first features of interest by the first stain or highlighting ofthe second features of interest by the second stain without highlightingof the first features of interest by the first stain. In some instances,the second features of interest may be different from the first featuresof interest.

In some embodiments, the second stain either may be the same as thefirst stain but used to stain the second features of interest or may bedifferent from the first stain. In some instances, the first stain andthe second stain may each include, but is not limited to, at least oneof Hematoxylin, Acridine orange, Bismarck brown, Carmine, Coomassieblue, Cresyl violet, Crystal violet, 4′,6-diamidino-2-phenylindole(“DAPI”), Eosin, Ethidium bromide intercalates, Acid fuchsine, Hoechststain, Iodine, Malachite green, Methyl green, Methylene blue, Neutralred, Nile blue, Nile red, Osmium tetroxide, Propidium Iodide, Rhodamine,Safranine, programmed death-ligand 1 (“PD-L1”) stain,3,3′-Diaminobenzidine (“DAB”) chromogen, Magenta chromogen, cyaninechromogen, cluster of differentiation (“CD”) 3 stain, CD20 stain, CD68stain, 40S ribosomal protein SA (“p40”) stain, antibody-based stain, orlabel-free imaging marker, and/or the like. In some cases, the firstimage 705 and the second image 710 may each include, without limitation,one of a set of color or brightfield images, a fluorescence image, or aspectral image, and/or the like, where the set of color or brightfieldimages may include, but is not limited to, a red-filtered (“R”) image, agreen-filtered (“G”) image, and a blue-filtered (“B”) image, and/or thelike.

As depicted in FIG. 7A (with the notation “WSI” being at a differentangle in the second image 710 compared with corresponding notation inthe first image 705), the first image 705 and the second image 710 mayinitially be misaligned, and thus may require alignment by alignmentsystem 715. The alignment system 715 may perform alignment of the firstimage 705 relative to the second image 710 (or vice versa) either usingmanual input from a user (e.g., pathologist or other operator, or thelike) or via autonomous alignment processes (in some cases, utilizing AIsystem functionality, or the like). In some cases, autonomous alignmentmay include, but is not limited to, alignment using at least one ofautomated global alignment techniques or optical flow alignmenttechniques, or the like.

In some embodiments, automated global alignment may include thefollowing computational details: (i) finding foreground tiles (in somecases, using entropy threshold criterion, or the like); (ii) performingfeature-based (e.g., with oriented fast and rotated brief (“ORB”)features, or the like) matching on low resolution images (e.g., defaultpyramid level 4, or the like); and (iii) computing affine alignmenttransformation from matched features. Computational steps (i) to (iii)are applicable to both serial and sequential staining. For sequentialstaining, the following additional details may apply for automatedglobal alignment: (iv) finding all high resolution (e.g., level 0) tilesspanning the matched low resolution features; (v) matching highresolution tiles based on low resolution alignment; (vi) performingfeature based matching on high resolution features within matched tiles;and (vii) computing global affine alignment as composition of the highand low resolution alignment transformation matrices; and/or the like.

Even with sequential staining, there may be misalignments anddeformations between scans. Global, affine alignment accounts fortranslation, rotation, or uniform deformation of the whole slide image,but cannot account for non-uniform deformation of the slide image. Largescale deformation may result in misalignment of paired FOVs, while smallscale deformation may result in non-uniform misalignment within a FOV(which may be apparent as horizontal stretching of the FOV, forexample). Such misalignment introduces noise to the training of modelsbased on the sequentially stained image as ground truth. Alignmentcharacterization may be optimized as follows: (a) optimization may beperformed on either H channel alone, or on the sum of H+DAB channels(combined); (b) optimization may be performed by non-overlapping squaremicro-patches of equal size having patch sides in 1000, 500, 100, or 50pixels in length; (c) manual global rigid alignment may be performed formicro-patch matching; (d) for each patch-pair, rigid transform (i.e.,rotation + translation) parameters may be found that minimize themis-match cost; (e) no continuity constraints across patches may beenforced - free optimization; (f) several cost functions may beevaluated by patch-level random self-alignment test, with the best (moststable) results being attained by pixel-wise MSE (L2) loss or the like;and (g) each micro-patch may be padded by adding extra pixels around thepatch to allow for required transformation range, which was not alwayspossible (due to FOV edges).

Quantitative evaluation of alignment characterization may alternativelyor additionally be performed as follows: (1) quantitative evaluation maybe performed by measuring cell-center distances in paired FOVS, withFOVs annotated using the predictions of segmentation models, withsegmentation net trained or retrained and running on the H channel only,and with centroids calculated using region_props from segmentationpredictions; (2) annotation in paired FOVs may be matched by k=1 nearestneighbour (“NN”) matching; (3) NN matching may be applied with thresholdon NN distance ratio (k=1, 2); (4) NN matching may also be applied withthreshold on a global distance threshold; and (5) matched distancehistograms pre and post-optimization may be generated for each FOV.Distance histograms pre and post optimization for micro-patches may beused for quantitative evaluation of the alignment characterization.

According to some embodiments, optical flow alignment may utilize denseinverse search (“DIS”), which may include the following characteristics:(A) optical flow dense (per-pixel) displacement field; (B) “coarse tofine” micro-patch based optimization with partial patch overlap; (C)utilizing an open-sourced implementation (e.g., Python bindings to a Cimplementation, or the like); (D) integrating into a computing platform(e.g., Matlab calling Python, or the like); (E) the algorithm’s patchsize and overlap may be empirically tuned for robustness over varioustissue samples and various image pyramid levels; and (F) achievinglatency per tile (including padding) of < 100 ms (in some evaluations,~40-50 ms was achieved). The algorithm details for optical flowalignment may include the following: (I) dividing the image into a gridof equally sized micro-patches; (II) for each micro-patch in the image,finding the displacement to its optimal position in the paired image(under MSE loss, or the like); (III) starting with coarse (lowresolution) patches, and using optimal displacement as the initialcondition for the next (finer) level; (IV) overlapping partialmicro-patch with similarity-weighted densification for continuity andsmoothness; and (V) applying variational refinement on intensityfunctions for further smoothness and conservation constraints (e.g.,divergence of brightness, color, and gradients, or the like). Distancehistograms pre and post optimization for DIS may be used forquantitative evaluation of the optical flow alignment. In summary, toaccount for local, non-uniform deformations, a second tier of localalignment -namely, optical flow alignment - may be used. Optical flowalignment refines the global alignment. Optical flow alignment canaccount for paired-FOV misalignment as well as non-uniform misalignmentwithin a FOV. With optical flow alignment, pixel-perfect registration ofthe sequentially stained images can be achieved.

In some cases, the received first and second images may be pre-alignedprior to being received, or may not require alignment (such as in thecase of output images of a fluorescence staining scanning protocol, orthe like), and thus alignment by the alignment system 715 may beomitted. After alignment (if needed), aligned image 720 may be used toextract at least one second image patch 725 (a plurality of which isdepicted in FIG. 7A). In some instances, aligned image 720 may bedivided into equal sized (and shaped) grids, with each image patch 725representing a grid. In some cases, the grids may be of different sizesand/or different shapes (e.g., rectangles, squares, trapezoids, circles,triangles, or other polygons, etc.).

Meanwhile, a nuclei detection system 730 (or other feature detectionsystem (not shown), or the like) may be used to detect nuclei (or otherfeatures of interest) in the first image 705, with the resultant image735 being used to train a second cell classifier net, a second AImodule, or a second model (“G*”; collectively, “second AI model G*” or“Model G*” or the like) 755. Unlike in the embodiment 400 of FIG. 4A,nuclei detection is not necessary for virtual staining, and thus atleast one first image patch 740 (a plurality of which is depicted inFIG. 7A) is extracted from the first image 705 rather than image 735, ina similar manner as the at least one second image patch 725. Each of theat least one first image patch 740 may correspond to each of the atleast one second image patch 725 to form patch pairs (although a set ofcorresponding three or more images, not limited to a pair, may be used,such as in the case of a third image (not shown) being used to highlighta third stain or a combination of stains).

The at least one first image patch 740 and the at least one second imagepatch 725 (or patch pairs, or the like) may serve as input images fortraining a first cell classifier net, a first AI module, or a first AImodel (“F”; collectively, “first AI model F” or the like) 745, which mayalso be referred to herein as a “virtual staining generator” or thelike. The first model G* 745 (which may, in some cases, receive externallabelling data, or the like, in addition to the nuclei or other featuredetection data, or the like) may generate a third patch or image 750comprising a virtual stain of the first aligned image patches, based atleast in part on the at least one first patch 725 and the at least onesecond patch 740, the virtual stain simulating staining by at least thesecond stain of features of interest in the first biological sample asshown in the at least one second patch 725.

The generated third patch or image 750, along with the image 735(containing results of the nuclei or other feature detection, or thelike) may be used as inputs to train the second AI model G 750 toidentify instances of features of interest in the first biologicalsample. The process flow as depicted in FIG. 7A represents acomputational pipeline (also referred to herein as “pipeline B”) forusing an AI model (in this case, model F) for generating virtualstaining for simulating sequentially stained biological sample, andusing such generated virtual staining to train another model (in thiscase, model G*) to identify features of interest in biological samples.

Turning to the non-limiting embodiment 700′of FIG. 7B, after the secondAI model G has been trained, the second AI model G 755′ may be used toreceive as an input, input image 760, and to identify features ofinterest in input image 760 and to generate an output 765 containinglabelling of identified features of interest in input image 760. In someembodiments, the output 765 may include, without limitation, at leastone of a clinical score and/or a labelled image (identifying features ofinterest), or the like. Such inferencing or overall scoring or diagnosiswould improve upon the technical field of annotation collection orautonomous annotation collection, by improving precision and accuracy inidentifying features of interest, avoiding observer/pathologist biasesor human error effects, increasing the speed of annotation collection,while, in some cases, identifying features of interest that aredifficult if not impossible for a human to identify. Additionally,pipeline B provides a simple, straight forward approach, with relativelylow complexity (in some cases, using binary classification, or thelike).

These and other features are similar, if not identical to the features,processes, and techniques described herein with respect to FIGS. 3-6, 8,and 9 , or the like.

FIGS. 7C and 7D may illustrate various process flows depicting virtualstaining performed by various embodiments of virtual stain net F or AImodel F 745 of FIG. 7A. Although Hematoxylin stain and Hematoxylin plusMagenta sequential staining are depicted in FIGS. 7C and 7D for theinput image 740 and the ground truth image 725, respectively, thevarious embodiments are not so limited, and the biological sampledepicted in the input image 740 and in the ground truth image 725 mayeach include, without limitation, at least one of Hematoxylin, Acridineorange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystalviolet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.

As shown in the non-limiting embodiment 700″ of FIG. 7C, one embodimentof virtual stain net F or AI model F 745′ may receive an input image 740(which may include, without limitation, an image of Hematoxylin stainedbiological sample, or the like) and ground truth image 725 (which mayinclude, without limitation, an image of Hematoxylin plus Magentasequentially stained biological sample, or the like), which are firstaligned (if not already aligned or if alignment is not necessary). Thevirtual stain net F or AI model F 745′ may include, but is not limitedto, an encoder 770 a, a decoder 770 b, an intensity map 770 c, anadditional convolutional neural network (“CNN”) 770 d (optional), acolor vector 770 e (e.g., an RGB vector or the like), a mixer 775 a, anadder 775 b, and a loss function device 780. An input image 740 is inputto the encoder/decoder network (e.g., the network of encoder 770 a anddecoder 770 b, or the like), which may be similar to a U-Net, with asingle channel output of the same spatial size as the input image (i.e.,output of Intensity Map 770 c). In some cases, the additional CNN 770 d(optional) may operate on the output of the encoder 770 a, with a colorvector output (i.e., output of RGB vector 770 e). The outputs of theintensity map 770 c and the color vector 770 e may be combined usingmixer 775 a to produce a 3-channel image of the virtual stain (i.e.,virtual magenta). The predicted virtual stain may be added using adder775 b to the input image 740 to produce the outputs - i.e., thepredicted virtually stained patch(es) or image(s) 750′. The network maythen be trained with a loss function (by the loss function device 780)between the virtually stained patch(es) or image(s) 750′ and its pairedsequentially stained ground truth image 725. In some embodiments, theloss function may include, but is not limited to, one of a mean squarederror loss function, a mean squared logarithmic error loss function, amean absolute error loss function, a Huber loss function, or a weightedsum of squared differences loss function, and/or the like.

Alternatively, as shown in the non-limiting embodiment 700‴ of FIG. 7D,another embodiment of virtual stain net F or AI model F 745″ may receivean input image 740 (which may include, without limitation, an image ofHematoxylin stained biological sample, or the like) and ground truthimage 725 (which may include, without limitation, an image ofHematoxylin plus Magenta sequentially stained biological sample, or thelike), which are first aligned (if not already aligned or if alignmentis not necessary). The virtual stain net F or AI model F 745″ mayinclude, but is not limited to, an AI model 785 and a loss functiondevice 780 (similar to loss function device 780 of FIG. 7C). An inputimage 740 is input to the AI model 785, which may output predictedvirtual stain may be added using adder 775 b′ to the input image 740 toproduce the outputs - i.e., the predicted virtually stained patch(es) orimage(s) 750″. The network may then be trained with a loss function (bythe loss function device 780) between the virtually stained patch(es) orimage(s) 750″ and its paired sequentially stained ground truth image725. In some embodiments, the loss function may include, but is notlimited to, one of a mean squared error loss function, a mean squaredlogarithmic error loss function, a mean absolute error loss function, aHuber loss function, or a weighted sum of squared differences lossfunction, and/or the like.

FIGS. 8A-8O (collectively, “FIG. 8 ”) are schematic diagramsillustrating various non-limiting examples 800 of images of biologicalsamples that have been virtually stained when implementing sequentialimaging of biological samples for generating training data forgenerating training data for developing deep learning based models forvirtual staining of biological samples and for developing deep learningbased models for image analysis, cell classification, and/or features ofinterest identification of biological samples, in accordance withvarious embodiments.

Although Hematoxylin plus DAB sequential staining and Hematoxylin plusDAB plus Magenta sequential staining are depicted in FIGS. 8A and 8B forthe input image and the ground truth image, respectively, or Hematoxylinstain and Hematoxylin plus p40 Magenta sequential staining are depictedin FIGS. 8C and 8D for the input image and the ground truth image,respectively, the various embodiments are not so limited, and thebiological sample depicted in the input image and in the ground truthimage may each include, without limitation, at least one of Hematoxylin,Acridine orange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet,Crystal violet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidiumbromide intercalates, Acid fuchsine, Hoechst stain, Iodine, Malachitegreen, Methyl green, Methylene blue, Neutral red, Nile blue, Nile red,Osmium tetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.

FIGS. 8A-8B depict a non-limiting example of an input image of a firstbiological sample having been stained with sequential staining ofHematoxylin stain and programmed death-ligand 1 (“PD-L1”) stain (e.g.,3,3′-Diaminobenzidine (“DAB”) chromogen, or the like) (as shown in FIG.8A) and a ground truth image of the first biological sample having beenstained with sequential staining of Hematoxylin stain, PD-L1 stain(e.g., DAB chromogen, or the like), and cluster of differentiation(“CD”) 68 stain (e.g., Magenta chromogen,) (as shown in FIG. 8B).

FIGS. 8C-8D depict a non-limiting example of an input image of a firstbiological sample having been stained with Hematoxylin stain (as shownin FIG. 8C) and a ground truth image of the first biological samplehaving been stained with sequential staining of Hematoxylin stain andp40 stain (e.g., Magenta chromogen,) (as shown in FIG. 8D).

FIG. 8E depicts an image of a biological sample with selected validationFOVs (denoted by blue rectangles).

Virtual Staining Modelling is depicted in FIGS. 8F-8I, with a magnifiedimage of a patch of the image containing the biological sample. FIG. 8Fdepicts an input image (and corresponding image patch) illustrating onlyHematoxylin stain, while FIG. 8G depicts a target image (andcorresponding image patch) illustrating sequential staining withHematoxylin stain and p40 (Magenta) stain, and FIG. 8H depicts unmixedMagenta stain intensity image (and corresponding image patch). FIG. 8Idepicts the Magenta stain intensity image of FIG. 8H, with an imagepatch that represents an intensity map, which can be used forregression, that is the prediction of a continuous value of the virtualstain intensity. FIG. 8I depicts a binarization of 8H that can be usedto train the network or model F as a classification network (binaryclassification). Training the classification network may requiredefining a threshold intensity value, beyond which a pixel will beconsidered a positive pixel for virtual stain. The question marks inFIG. 8I (i.e., “classification?” and “threshold?”) are indicative of thepossibility of solving this as a classification problem and/or therequirement of determining a threshold if doing so. In some non-limitingexample models that have been trained, a regression solution wasselected that solved for the kind of intensity image shown in FIG. 8I,after combining it with the color vector and adding to the input image.

FIGS. 8J-8O depict evaluation of the virtual staining technique (in thiscase, p40 nucleic virtual staining (FIGS. 8J-8L) and CD3 virtualstaining (FIGS. 8M-8O)). FIGS. 8J and 8K depict an originally stainedimage of a biological sample and a virtually stained image of thebiological sample, respectively, both at 40X magnification, while FIG.8L depicts a color-coded evaluation of the virtually stained image ofFIG. 8K, with true positive nuclei detection denoted by blue dots, falsepositive nuclei detection denoted by red dots, true negative nucleidetection denoted by green dots, and false negative nuclei detectiondenoted by black dots. FIGS. 8N and 8O similarly depict color-codedevaluation of the virtually stained image of FIG. 8M, with true positivenuclei detection denoted by blue dots, false positive nuclei detectiondenoted by red dots, true negative nuclei detection denoted by greendots, and false negative nuclei detection denoted by black dots.

FIGS. 9A-9E (collectively, “FIG. 9 ”) are flow diagrams illustrating amethod 900 for implementing sequential imaging of biological samples forgenerating training data for generating training data for developingdeep learning based models for virtual staining of biological samplesand for developing deep learning based models for image analysis, cellclassification, and/or features of interest identification of biologicalsamples, in accordance with various embodiments.

Method 900 of FIG. 9A continues onto FIG. 9B following the circularmarker denoted, “A.”

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 900 illustrated byFIG. 9 can be implemented by or with (and, in some cases, are describedbelow with respect to) the systems, examples, or embodiments 300, 700,700′, 700″, 700‴, and 800 of FIGS. 3, 7A, 7B, 7C, 7D, and 8 ,respectively (or components thereof), such methods may also beimplemented using any suitable hardware (or software) implementation.Similarly, while each of the systems, examples, or embodiments 300, 700,700′, 700″, 700‴, and 800 of FIGS. 3, 7A, 7B, 7C, 7D, and 8 ,respectively (or components thereof), can operate according to themethod 900 illustrated by FIG. 9 (e.g., by executing instructionsembodied on a computer readable medium), the systems, examples, orembodiments 300, 700, 700′, 700″, 700‴, and 800 of FIGS. 3, 7A, 7B, 7C,7D, and 8 can each also operate according to other modes of operationand/or perform other suitable procedures.

In the non-limiting embodiment of FIG. 9A, method 900, at block 905, maycomprise receiving, with a computing system, a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample that has been stained with afirst stain. Method 900 may comprise receiving, with the computingsystem, a second image of the first biological sample, the second imagecomprising a second FOV of the first biological sample that has beenstained with at least a second stain (block 910).

At block 915, method 900 may comprise aligning, with the computingsystem, the first image with the second image to create first alignedimages, by aligning one or more features of interest in the firstbiological sample as depicted in the first image with the same one ormore features of interest in the first biological sample as depicted inthe second image. According to some embodiments, aligning the firstimage with the second image may be either performed manually usingmanual inputs to the computing system or performed autonomously, whereautonomous alignment may comprise alignment using at least one ofautomated global alignment techniques or optical flow alignmenttechniques.

Method 900 may further comprise, at block 920, autonomously creating,with the computing system, first aligned image patches from the firstaligned images, by extracting a portion of the first aligned images. Insome cases, the portion of the first aligned images may comprise a firstpatch corresponding to the extracted portion of the first image and asecond patch corresponding to the extracted portion of the second image.

Method 900 may further comprise training, using an artificialintelligence (“AI”) system, a first AI model (“Model F”) to generate athird patch comprising a virtual stain of the first aligned imagepatches, based at least in part on the first patch and the second patch,the virtual stain simulating staining by at least the second stain offeatures of interest in the first biological sample as shown in thesecond patch (block 925); and training, using the AI system, a secondmodel (“Model G*”) to identify or classify first instances of featuresof interest in the first biological sample, based at least in part onthe third patch and based at least in part on results from an externalinstance classification process or a region of interest detectionprocess (block 730).

In some embodiments, training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample may comprise training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample, based at least in part on one or more of the first patch, thesecond patch, or the third patch and based at least in part on theresults from the external instance classification process or the regionof interest detection process. According to some embodiments, theexternal instance classification process or the region of interestdetection process may each include, without limitation, at least one ofdetection of nuclei in the first image or the first patch by a nucleidetection method, identification of nuclei in the first image or thefirst patch by a pathologist, detection of features of interest in thefirst image or the first patch by a feature detection method, oridentification of features of interest in the first image or the firstpatch by the pathologist, and/or the like.

In some embodiments, the computing system may include, withoutlimitation, one of a computing system disposed in a work environment, aremote computing system disposed external to the work environment andaccessible over a network, a web server, a web browser, or a cloudcomputing system, and/or the like. According to some embodiments, the AIsystem may include, but is not limited to, at least one of a machinelearning system, a deep learning system, a model architecture, astatistical model-based system, or a deterministic analysis system,and/or the like. In some cases, the model architecture may comprise atleast one of a neural network, a convolutional neural network (“CNN”),or a fully convolutional network (“FCN”), and/or the like. In someinstances, the work environment may include, but is not limited to, atleast one of a laboratory, a clinic, a medical facility, a researchfacility, a healthcare facility, or a room, and/or the like. In somecases, the first biological sample may include, without limitation, oneof a human tissue sample, an animal tissue sample, a plant tissuesample, or an artificially produced tissue sample, and/or the like, andthe features of interest may include, but are not limited to, at leastone of normal cells, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, or organ structures, and/or the like.

Merely by way of example, in some cases, the first image may comprisehighlighting of first features of interest in the first biologicalsample (including, but not limited to, at least one of first antigens,first nuclei, first cell walls, first cell structures, first antibodies,first normal cells, first abnormal cells, first damaged cells, firstcancer cells, first tumors, first subcellular structures, first organstructures, or other features of interest, or the like) by the firststain that had been applied to the first biological sample. The thirdpatch may comprise one of highlighting of the first features of interestby the first stain and highlighting of second features of interest inthe first biological sample (including, but not limited to, at least oneof second antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the virtual stain that simulates the secondstain having been applied to the first biological sample or highlightingof the first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest. In someinstances, the second stain may be one of the same as the first stainbut used to stain the second features of interest or different from thefirst stain.

According to some embodiments, the first stain and the second stain mayeach include, without limitation, at least one of Hematoxylin, Acridineorange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystalviolet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.

In some embodiments, the first image may include, but is not limited to,one of a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like.The first set of color or brightfield images may include, withoutlimitation, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. Similarly, the second image may include, but is notlimited to, one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like. The second set of color or brightfield images mayinclude, without limitation, a second R image, a second G image, and asecond B image, and/or the like. The second fluorescence image mayinclude, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike. Likewise, the third patch may include, but is not limited to, oneof a third set of color or brightfield images, a third fluorescenceimage, a third phase image, or a third spectral image, and/or the like.The third set of color or brightfield images may include, withoutlimitation, a third R image, a third G image, and a third B image,and/or the like. The third fluorescence image may include, but is notlimited to, at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may include, but is not limited to, at least one of athird Raman spectroscopy image, a third NIR spectroscopy image, a thirdmultispectral image, a third hyperspectral image, or a third fullspectral image, and/or the like.

Method 900 may continue onto the process at block 935 in FIG. 9Bfollowing the circular marker denoted, “A.”

At block 935 in FIG. 9B (following the circular marker denoted, “A”),method 900 may comprise receiving, with the computing system, a fourthimage, the fourth image comprising one of a fourth FOV of the firstbiological sample different from the first FOV and the second FOV or afifth FOV of a second biological sample, where the second biologicalsample is different from the first biological sample. Method 900 mayfurther comprise identifying, using Model G*, second instances offeatures of interest in the second biological sample, based at least inpart on the fourth image and based at least in part on training of ModelG* using at least the third patch comprising the virtual stain of thefirst aligned image patches (block 940). According to some embodiments,method 900 may further comprise generating, using Model G*, a clinicalscore, based at least in part on the identified second instances offeatures of interest.

Turning to FIG. 9C, method 900 may comprise, at block 945, receiving,with a computing system, a first image of a first biological sample, thefirst image comprising a first field of view (“FOV”) of the firstbiological sample. At block 950, method 900 may comprise identifying,using a first model (“Model G*”) that is trained by an artificialintelligence (“AI”) system, first instances of features of interest inthe first biological sample, based at least in part on the first imageand based at least in part on training of Model G* using at least afirst patch comprising a virtual stain of first aligned image patches.In some cases, the first patch may be generated by a second AI model(Model F) that is generated or updated by the trained AI system by usinga second patch. The first aligned image patches may comprise anextracted portion of first aligned images. The first aligned images maycomprise a second image and a third image that have been aligned. Thesecond image may comprise a second FOV of a second biological samplethat is different from the first biological sample that has been stainedwith a first stain. The second patch may comprise the extracted portionof the second image. The third image may comprise a third FOV of thesecond biological sample that has been stained with at least a secondstain.

With reference to FIGS. 9D and 9E, training the AI system to updateModel F may comprise (as shown in FIG. 9D): receiving, with an encoder,the first patch (block 925a); receiving the second patch (block 925b);encoding, with the encoder, the received first patch (block 925c);decoding, with the decoder, the encoded first patch (block 925d);generating an intensity map based on the decoded first patch (block925e); simultaneously operating on the encoded first patch to generate acolor vector (block 925f); combining the generated intensity map withthe generated color vector to generate an image of the virtual stain(block 925g); generating a predicted virtually stained image patch(block 925h); determining a first loss value between the predictedvirtually stained image patch and the second patch (block 925i);calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch (block 925j); and updating, using the AI system, Model F togenerate the third patch, by updating one or more parameters of Model Fbased on the calculated loss value (block 925k). According to someembodiments, generating the predicted virtually stained image patch (atblock 925h) may comprise adding the generated image of the virtual stainto the received first patch to produce the predicted virtually stainedimage patch. In some cases, determining the first loss value between thepredicted virtually stained image patch and the second patch may beperformed without adding the generated image of the virtual stain to thereceived first patch.

In some embodiments, the loss function may include, without limitation,one of a mean squared error loss function, a mean squared logarithmicerror loss function, a mean absolute error loss function, a Huber lossfunction, or a weighted sum of squared differences loss function, and/orthe like.

In some cases, the first patch may include, but is not limited to, oneof a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like.The first set of color or brightfield images may include, withoutlimitation, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. The second image may include, but is not limited to,one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like. The second set of color or brightfield images mayinclude, without limitation, a second R image, a second G image, and asecond B image, and/or the like. The second fluorescence image mayinclude, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike. The predicted virtually stained image patch may include, but isnot limited to, one of a third set of color or brightfield images, athird fluorescence image, a third phase image, or a third spectralimage, and/or the like. The third set of color or brightfield images mayinclude, without limitation, a third R image, a third G image, and athird B image, and/or the like. The third fluorescence image mayinclude, but is not limited to, at least one of a third autofluorescenceimage or a third labelled fluorescence image having one or more channelswith different excitation or emission characteristics, and/or the like.The third spectral image may include, but is not limited to, at leastone of a third Raman spectroscopy image, a third NIR spectroscopy image,a third multispectral image, a third hyperspectral image, or a thirdfull spectral image, and/or the like.

In some instances, operating on the encoded first patch to generate thecolor vector may comprise operating on the encoded first patch togenerate a color vector, using a color vector output and one of a modelarchitecture, a statistical model-based system, or a deterministicanalysis system, and/or the like. In some instances, the modelarchitecture may comprise at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”), and/or the like. The color vector may include, withoutlimitation, one of a fixed color vector or a learned color vector thatis not based on the encoded first patch, and/or the like. In some cases,the image of the virtual stain may include, but is not limited to, oneof a 3-channel image of the virtual stain, a RGB-transform image of thevirtual stain, or a logarithmic transform image of the virtual stain,and/or the like. In some embodiments, when the virtual stain is thelogarithmic transform of the virtual stain, it may be added to thelogarithmic transform of the input image (i.e., the first patch). Afterthe addition, the logarithmically transformed virtually stained image istransformed, using an exponential transform, back to the common domainof RGB images (i.e., the reverse transform of the logarithmictransform). In such a case, the loss during training may be computed onthe logarithmically transformed virtually stained image (or logtransformed virtual stain directly in some cases), compared to the logtransformed second patch (e.g., sequentially stained magenta,fluorescence, etc.). In some cases, there may be no exponentiationbefore calculating the loss, but the loss can also be calculated on thecommon RGB images domain, after exponentiating the virtually stainedimage and comparing to the second patch directly (i.e., notlog-transformed).

According to some embodiments, the first loss value may include, withoutlimitation, one of a pixel loss value between each pixel in thepredicted virtually stained image patch and a corresponding pixel in thesecond patch or a generative adversarial network (“GAN”) loss valuebetween the predicted virtually stained image patch and the secondpatch, and/or the like. In some instances, the GAN loss value may begenerated based on one of a minimax GAN loss function, a non-saturatingGAN loss function, a least squares GAN loss function, or a WassersteinGAN loss function, and/or the like.

Alternatively, training the AI system to update Model F may comprise (asshown in FIG. 9E): receiving, with the AI system, the first patch (block9251); receiving, with the AI system, the second patch (block 925m);generating, with a second model of the AI system, an image of thevirtual stain (block 925n); adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch (block 925o); determining a first loss value betweenthe predicted virtually stained image patch and the second patch (block925p); calculating a loss value using a loss function, based on thefirst loss value between the predicted virtually stained image patch andthe second patch (block 925q); and updating, with the AI system, Model Fto generate the third patch, by updating one or more parameters of ModelF based on the calculated loss value (block 925r).

In some embodiments, the second model may include, without limitation,at least one of a convolutional neural network (“CNN”), a U-Net, anartificial neural network (“ANN”), a residual neural network (“ResNet”),an encode/decode CNN, an encode/decode U-Net, an encode/decode ANN, oran encode/decode ResNet, and/or the like.

Exemplary System and Hardware Implementation

FIG. 10 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments. FIG. 10provides a schematic illustration of one embodiment of a computer system1000 of the service provider system hardware that can perform themethods provided by various other embodiments, as described herein,and/or can perform the functions of computer or hardware system (i.e.,computing systems 305 a and 305 b, artificial intelligence (“AI”)systems 305 c and 305 d and/or AI system architecture 445, 445′, 450,450′, 745, 745′, 745″, 755, and 755′, display device 320, user device(s)340, image alignment systems 415 and 715, and nuclei detection systems430 and 730, etc.), as described above. It should be noted that FIG. 10is meant only to provide a generalized illustration of variouscomponents, of which one or more (or none) of each may be utilized asappropriate. FIG. 10 , therefore, broadly illustrates how individualsystem elements may be implemented in a relatively separated orrelatively more integrated manner.

The computer or hardware system 1000 - which might represent anembodiment of the computer or hardware system (i.e., computing systems305 a and 305 b, AI systems 305 c and 305 d and/or AI systemarchitecture 445, 445′, 450, 450′, 745, 745′, 745″, 755, and 755′,display device 320, user device(s) 340, image alignment systems 415 and715, and nuclei detection systems 430 and 730, etc.), described abovewith respect to FIGS. 3-9 -is shown comprising hardware elements thatcan be electrically coupled via a bus 1005 (or may otherwise be incommunication, as appropriate). The hardware elements may include one ormore processors 1010, including, without limitation, one or moregeneral-purpose processors and/or one or more special-purpose processors(such as microprocessors, digital signal processing chips, graphicsacceleration processors, and/or the like); one or more input devices1015, which can include, without limitation, a mouse, a keyboard, and/orthe like; and one or more output devices 1020, which can include,without limitation, a display device, a printer, and/or the like.

The computer or hardware system 1000 may further include (and/or be incommunication with) one or more storage devices 1025, which cancomprise, without limitation, local and/or network accessible storage,and/or can include, without limitation, a disk drive, a drive array, anoptical storage device, solid-state storage device such as a randomaccess memory (“RAM”) and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable, and/or the like. Such storage devicesmay be configured to implement any appropriate data stores, including,without limitation, various file systems, database structures, and/orthe like.

The computer or hardware system 1000 might also include a communicationssubsystem 1030, which can include, without limitation, a modem, anetwork card (wireless or wired), an infra-red communication device, awireless communication device and/or chipset (such as a Bluetooth®device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device,cellular communication facilities, etc.), and/or the like. Thecommunications subsystem 1030 may permit data to be exchanged with anetwork (such as the network described below, to name one example), withother computer or hardware systems, and/or with any other devicesdescribed herein. In many embodiments, the computer or hardware system1000 will further comprise a working memory 1035, which can include aRAM or ROM device, as described above.

The computer or hardware system 1000 also may comprise softwareelements, shown as being currently located within the working memory1035, including an operating system 1040, device drivers, executablelibraries, and/or other code, such as one or more application programs1045, which may comprise computer programs provided by variousembodiments (including, without limitation, hypervisors, VMs, and thelike), and/or may be designed to implement methods, and/or configuresystems, provided by other embodiments, as described herein. Merely byway of example, one or more procedures described with respect to themethod(s) discussed above might be implemented as code and/orinstructions executable by a computer (and/or a processor within acomputer); in an aspect, then, such code and/or instructions can be usedto configure and/or adapt a general purpose computer (or other device)to perform one or more operations in accordance with the describedmethods.

A set of these instructions and/or code might be encoded and/or storedon a non-transitory computer readable storage medium, such as thestorage device(s) 1025 described above. In some cases, the storagemedium might be incorporated within a computer system, such as thesystem 1000. In other embodiments, the storage medium might be separatefrom a computer system (i.e., a removable medium, such as a compactdisc, etc.), and/or provided in an installation package, such that thestorage medium can be used to program, configure, and/or adapt a generalpurpose computer with the instructions/code stored thereon. Theseinstructions might take the form of executable code, which is executableby the computer or hardware system 1000 and/or might take the form ofsource and/or installable code, which, upon compilation and/orinstallation on the computer or hardware system 1000 (e.g., using any ofa variety of generally available compilers, installation programs,compression/decompression utilities, etc.) then takes the form ofexecutable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware (such as programmable logic controllers,field-programmable gate arrays, application-specific integratedcircuits, and/or the like) might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer or hardware system (such as the computer or hardware system1000) to perform methods in accordance with various embodiments of theinvention. According to a set of embodiments, some or all of theprocedures of such methods are performed by the computer or hardwaresystem 1000 in response to processor 1010 executing one or moresequences of one or more instructions (which might be incorporated intothe operating system 1040 and/or other code, such as an applicationprogram 1045) contained in the working memory 1035. Such instructionsmay be read into the working memory 1035 from another computer readablemedium, such as one or more of the storage device(s) 1025. Merely by wayof example, execution of the sequences of instructions contained in theworking memory 1035 might cause the processor(s) 1010 to perform one ormore procedures of the methods described herein.

The terms “machine readable medium” and “computer readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In an embodimentimplemented using the computer or hardware system 1000, various computerreadable media might be involved in providing instructions/code toprocessor(s) 1010 for execution and/or might be used to store and/orcarry such instructions/code (e.g., as signals). In manyimplementations, a computer readable medium is a non-transitory,physical, and/or tangible storage medium. In some embodiments, acomputer readable medium may take many forms, including, but not limitedto, non-volatile media, volatile media, or the like. Non-volatile mediaincludes, for example, optical and/or magnetic disks, such as thestorage device(s) 1025. Volatile media includes, without limitation,dynamic memory, such as the working memory 1035. In some alternativeembodiments, a computer readable medium may take the form oftransmission media, which includes, without limitation, coaxial cables,copper wire, and fiber optics, including the wires that comprise the bus1005, as well as the various components of the communication subsystem1030 (and/or the media by which the communications subsystem 1030provides communication with other devices). In an alternative set ofembodiments, transmission media can also take the form of waves(including without limitation radio, acoustic, and/or light waves, suchas those generated during radio-wave and infra-red data communications).

Common forms of physical and/or tangible computer readable mediainclude, for example, a floppy disk, a flexible disk, a hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punch cards, paper tape, any other physical medium with patternsof holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, a carrier wave as described hereinafter, or any othermedium from which a computer can read instructions and/or code.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 1010for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer or hardware system 1000. Thesesignals, which might be in the form of electromagnetic signals, acousticsignals, optical signals, and/or the like, are all examples of carrierwaves on which instructions can be encoded, in accordance with variousembodiments of the invention.

The communications subsystem 1030 (and/or components thereof) generallywill receive the signals, and the bus 1005 then might carry the signals(and/or the data, instructions, etc. carried by the signals) to theworking memory 1035, from which the processor(s) 1005 retrieves andexecutes the instructions. The instructions received by the workingmemory 1035 may optionally be stored on a storage device 1025 eitherbefore or after execution by the processor(s) 1010.

As noted above, a set of embodiments comprises methods and systems forimplementing annotation data collection and autonomous annotation, and,more particularly, to methods, systems, and apparatuses for implementingsequential imaging of biological samples for generating training datafor developing deep learning based models for image analysis, for cellclassification, for feature of interest identification, and/or forvirtual staining of biological samples. FIG. 11 illustrates a schematicdiagram of a system 1100 that can be used in accordance with one set ofembodiments. The system 1100 can include one or more user computers,user devices, or customer devices 1105. A user computer, user device, orcustomer device 1105 can be a general purpose personal computer(including, merely by way of example, desktop computers, tabletcomputers, laptop computers, handheld computers, and the like, runningany appropriate operating system, several of which are available fromvendors such as Apple, Microsoft Corp., and the like), cloud computingdevices, a server(s), and/or a workstation computer(s) running any of avariety of commercially-available UNIX® or UNIX-like operating systems.A user computer, user device, or customer device 1105 can also have anyof a variety of applications, including one or more applicationsconfigured to perform methods provided by various embodiments (asdescribed above, for example), as well as one or more officeapplications, database client and/or server applications, and/or webbrowser applications. Alternatively, a user computer, user device, orcustomer device 1105 can be any other electronic device, such as athin-client computer, Internet-enabled mobile telephone, and/or personaldigital assistant, capable of communicating via a network (e.g., thenetwork(s) 1110 described below) and/or of displaying and navigating webpages or other types of electronic documents. Although the exemplarysystem 1100 is shown with two user computers, user devices, or customerdevices 1105, any number of user computers, user devices, or customerdevices can be supported.

Certain embodiments operate in a networked environment, which caninclude a network(s) 1110. The network(s) 1110 can be any type ofnetwork familiar to those skilled in the art that can support datacommunications using any of a variety of commercially-available (and/orfree or proprietary) protocols, including, without limitation, TCP/IP,SNA®, IPX®, AppleTalk®, and the like. Merely by way of example, thenetwork(s) 1110 (similar to network(s) 350 of FIG. 3 , or the like) caneach include a local area network (“LAN”), including, withoutlimitation, a fiber network, an Ethernet network, a Token-Ring® network,and/or the like; a wide-area network (“WAN”); a wireless wide areanetwork (“WWAN”); a virtual network, such as a virtual private network(“VPN”); the Internet; an intranet; an extranet; a public switchedtelephone network (“PSTN”); an infra-red network; a wireless network,including, without limitation, a network operating under any of the IEEE802.11 suite of protocols, the Bluetooth® protocol known in the art,and/or any other wireless protocol; and/or any combination of theseand/or other networks. In a particular embodiment, the network mightinclude an access network of the service provider (e.g., an Internetservice provider (“ISP”)). In another embodiment, the network mightinclude a core network of the service provider, and/or the Internet.

Embodiments can also include one or more server computers 1115. Each ofthe server computers 1115 may be configured with an operating system,including, without limitation, any of those discussed above, as well asany commercially (or freely) available server operating systems. Each ofthe servers 1115 may also be running one or more applications, which canbe configured to provide services to one or more clients 1105 and/orother servers 1115.

Merely by way of example, one of the servers 1115 might be a dataserver, a web server, a cloud computing device(s), or the like, asdescribed above. The data server might include (or be in communicationwith) a web server, which can be used, merely by way of example, toprocess requests for web pages or other electronic documents from usercomputers 1105. The web server can also run a variety of serverapplications, including HTTP servers, FTP servers, CGI servers, databaseservers, Java servers, Matlab servers, and the like. In some embodimentsof the invention, the web server may be configured to serve web pagesthat can be operated within a web browser on one or more of the usercomputers 1105 to perform methods of the invention.

The server computers 1115, in some embodiments, might include one ormore application servers, which can be configured with one or moreapplications accessible by a client running on one or more of the clientcomputers 1105 and/or other servers 1115. Merely by way of example, theserver(s) 1115 can be one or more general purpose computers capable ofexecuting programs or scripts in response to the user computers 1105and/or other servers 1115, including, without limitation, webapplications (which might, in some cases, be configured to performmethods provided by various embodiments). Merely by way of example, aweb application can be implemented as one or more scripts or programswritten in any suitable programming language, such as Java™®, C, C#™ orC++, and/or any scripting language, such as Perl, Python, or TCL, aswell as combinations of any programming and/or scripting languages. Theapplication server(s) can also include database servers, including,without limitation, those commercially available from Oracle®,Microsoft®, Sybase®, IBM®, and the like, which can process requests fromclients (including, depending on the configuration, dedicated databaseclients, API clients, web browsers, etc.) running on a user computer,user device, or customer device 1105 and/or another server 1115. In someembodiments, an application server can perform one or more of theprocesses for implementing annotation data collection and autonomousannotation, and, more particularly, to methods, systems, and apparatusesfor implementing sequential imaging of biological sample for generatingtraining data for developing deep learning based models for imageanalysis, for cell classification, for feature of interestidentification, and/or for virtual staining of biological samples, asdescribed in detail above. Data provided by an application server may beformatted as one or more web pages (comprising HTML, JavaScript, etc.,for example) and/or may be forwarded to a user computer 1105 via a webserver (as described above, for example). Similarly, a web server mightreceive web page requests and/or input data from a user computer 1105and/or forward the web page requests and/or input data to an applicationserver. In some cases, a web server may be integrated with anapplication server.

In accordance with further embodiments, one or more servers 1115 canfunction as a file server and/or can include one or more of the files(e.g., application code, data files, etc.) necessary to implementvarious disclosed methods, incorporated by an application running on auser computer 1105 and/or another server 1115. Alternatively, as thoseskilled in the art will appreciate, a file server can include allnecessary files, allowing such an application to be invoked remotely bya user computer, user device, or customer device 1105 and/or server1115.

It should be noted that the functions described with respect to variousservers herein (e.g., application server, database server, web server,file server, etc.) can be performed by a single server and/or aplurality of specialized servers, depending on implementation-specificneeds and parameters.

In certain embodiments, the system can include one or more databases1120a-1120n (collectively, “databases 1120”). The location of each ofthe databases 1120 is discretionary: merely by way of example, adatabase 1120a might reside on a storage medium local to (and/orresident in) a server 1115a (and/or a user computer, user device, orcustomer device 1105). Alternatively, a database 1120n can be remotefrom any or all of the computers 1105, 1115, so long as it can be incommunication (e.g., via the network 1110) with one or more of these. Ina particular set of embodiments, a database 1120 can reside in astorage-area network (“SAN”) familiar to those skilled in the art.(Likewise, any necessary files for performing the functions attributedto the computers 1105, 1115 can be stored locally on the respectivecomputer and/or remotely, as appropriate.) In one set of embodiments,the database 1120 can be a relational database, such as an Oracledatabase, that is adapted to store, update, and retrieve data inresponse to SQL-formatted commands. The database might be controlledand/or maintained by a database server, as described above, for example.

According to some embodiments, system 1100 might further comprise acomputing system 1125 (similar to computing systems 305 a of FIG. 3 , orthe like) and corresponding database(s) 1130 (similar to database(s) 310a of FIG. 3 , or the like). System 1100 might further comprise a displaydevice 1140 (similar to display device 320 of FIG. 3 , or the like) thatare used to allow a user 1145 to look at an optical view of a firstbiological sample that is displayed on the display device 1140. The user1145 might use one or more user devices 1160 (similar to user device(s)340 of FIG. 3 , or the like; including, without limitation, smartphones, mobile phones, tablet computers, laptop computers, desktopcomputers, keyboards, keypads, computer mice, or monitors, and/or thelike). In some embodiments, system 1100 might further comprise one ormore audio sensors 1155 (optional; similar to audio sensor(s) 335 ofFIG. 3 , or the like; including, but not limited to, one or moremicrophones, one or more voice recorders, or one or more audiorecorders, and/or the like), a camera 1150 (optional; similar to camera330 of FIG. 3 , or the like; including, without limitation, one or moreeye tracking sensors, one or more motion sensors, or one or moretracking sensors, and/or the like), and a microscope 1135 (optional;similar to microscopes 315 of FIG. 3 , or the like). In some cases, theaudio sensors 1155 might be used to record vocal or spoken annotationsby the user 1145 while the user is viewing the FOV of the firstbiological sample either on the display device 1140 or through aneyepiece(s) of the microscope 1135. The camera 1150 might capture imagesof the user 1145 (in some cases, capturing images of at least one eye ofthe user 1145) while the user 1145 is within the field of view (“FOV”)1150a of camera 1150, as the user is viewing the FOV of the firstbiological sample either on the display device 1140 or through aneyepiece(s) of the microscope 1135. The features of gaze tracking andvoice annotation are described in greater detail in the ‘105Application, which has already been incorporated by reference in itsentirety for all purposes. In some instances, two or more of computingsystem 1125, database(s) 1130, display device 1140, user device(s) 1160,audio sensor(s) 1155 (optional), camera 1150 (optional), and/ormicroscope 1135 (optional) might be disposed in work environment 1165,which might include, but is not limited to, at least one of alaboratory, a clinic, a medical facility, a research facility, ahealthcare facility, or a room, and/or the like.

Alternative, or additional, to computing system 1125 and correspondingdatabase(s) 1130, system 1100 might further comprise remote computingsystem 1170 (similar to remote computing system 305 b of FIG. 3 , or thelike) and corresponding database(s) 1175 (similar to database(s) 310 bof FIG. 3 , or the like). In some embodiments, system 1100 might furthercomprise artificial intelligence (“AI”) system 1180. In someembodiments, computing system 1125 and/or 1170 might include, withoutlimitation, one of a computing system disposed in a work environment, aremote computing system disposed external to the work environment andaccessible over a network, a web server, a web browser, or a cloudcomputing system, and/or the like. According to some embodiments, the AIsystem 1180 might include, but is not limited to, at least one of amachine learning system, a deep learning system, a model architecture,and/or the like. In some instances, the model architecture may compriseat least one of a neural network, a convolutional neural network(“CNN”), or a fully convolutional network (“FCN”), and/or the like.

In operation, computing system 1125, remote computing system 1170,and/or AI system 1180 (collectively, “computing system”) may receive afirst image of a first biological sample, the first image comprising afirst field of view (“FOV”) of the first biological sample that has beenstained with a first stain; and may receive a second image of the firstbiological sample, the second image comprising a second FOV of the firstbiological sample that has been stained with a second stain. Thecomputing system may autonomously create a first set of image patchesbased on the first image and the second image, by extracting a portionof the first image and extracting a corresponding portion of the secondimage, the first set of image patches comprising a first patchcorresponding to the extracted portion of the first image and a secondpatch corresponding to the extracted portion of the second image. Insuch cases, the first set of image patches may comprise labeling ofinstances of features of interest in the first biological sample that isbased at least in part on at least one of information contained in thefirst patch, information contained in the second patch, or informationcontained in one or more external labeling sources.

The computing system utilize an AI system (e.g., AI system 305 d or 305c, or the like) to train a first model (“Model G*”) to generate firstinstance classification of features of interest (“Ground Truth”) in thefirst biological sample, based at least in part on the first set ofimage patches and the labeling of instances of features of interestcontained in the first set of image patches; and may utilize the AIsystem to train a second AI model (“Model G”) to identify instances offeatures of interest in the first biological sample, based at least inpart on the first patch and the first instance classification offeatures of interest generated by Model G*.

According to some embodiments, the first biological sample may include,without limitation, one of a human tissue sample, an animal tissuesample, a plant tissue sample, or an artificially produced tissuesample, and/or the like. In some instances, the features of interest mayinclude, but are not limited to, at least one of normal cells, abnormalcells, damaged cells, cancer cells, tumors, subcellular structures, ororgan structures, and/or the like.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample (including, but notlimited to, at least one of first antigens, first nuclei, first cellwalls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. Similarly, the second image maycomprise one of highlighting of second features of interest in the firstbiological sample (including, but not limited to, at least one of secondantigens, second nuclei, second cell walls, second cell structures,second antibodies, second normal cells, second abnormal cells, seconddamaged cells, second cancer cells, second tumors, second subcellularstructures, second organ structures, or other features of interest, orthe like) by the second stain that had been applied to the firstbiological sample in addition to highlighting of the first features ofinterest by the first stain or highlighting of the second features ofinterest by the second stain without highlighting of the first featuresof interest by the first stain, the second features of interest beingdifferent from the first features of interest. In some cases, the secondstain may be one of the same as the first stain but used to stain thesecond features of interest or different from the first stain, or thelike.

Merely by way of example, in some cases, the first stain may include,without limitation, at least one of Hematoxylin, Acridine orange,Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystal violet,4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.Likewise, the second stain may include, but is not limited to, at leastone of Hematoxylin, Acridine orange, Bismarck brown, Carmine, Coomassieblue, Cresyl violet, Crystal violet, DAPI, Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, PD-L1 stain, DABchromogen, Magenta chromogen, cyanine chromogen, CD3 stain, CD20 stain,CD68 stain, p40 stain, antibody-based stain, or label-free imagingmarker, and/or the like.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image or a firstlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. Similarly, thesecond image may include, without limitation, one of a second set ofcolor or brightfield images, a second fluorescence image, a second phaseimage, or a second spectral image, and/or the like, where the second setof color or brightfield images may include, but is not limited to, asecond R image, a second G image, and a second B image, and/or the like.The second fluorescence image may include, but is not limited to, atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The second spectral imagemay include, but is not limited to, at least one of a second Ramanspectroscopy image, a second NIR spectroscopy image, a second multispectral image, a second hyperspectral image, or a second full spectralimage, and/or the like.

In some embodiments, the computing system may align the first image withthe second image to create first set of aligned images, by aligning oneor more features of interest in the first biological sample as depictedin the first image with the same one or more features of interest in thefirst biological sample as depicted in the second image. In someinstances, aligning the first image with the second image may be one ofperformed manually using manual inputs to the computing system orperformed autonomously, where autonomous alignment may comprisealignment using at least one of automated global alignment techniques oroptical flow alignment techniques, or the like.

According to some embodiments, the computing system may receive a thirdimage of the first biological sample, the third image comprising a thirdFOV of the first biological sample that has been stained with a thirdstain different from each of the first stain and the second stain; andmay autonomously perform one of: aligning the third image with each ofthe first image and the second image to create second aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the third image with the same one or more features ofinterest in the first biological sample as depicted in each of the firstimage and the second image; or aligning the third image with the firstaligned images to create second aligned images, by aligning one or morefeatures of interest in the first biological sample as depicted in thethird image with the same one or more features of interest in the firstbiological sample as depicted in the first aligned images.

The computing system may autonomously create second aligned imagepatches from the second aligned images, by extracting a portion of thesecond aligned images, the portion of the second aligned imagescomprising a first patch corresponding to the extracted portion of thefirst image, a second patch corresponding to the extracted portion ofthe second image, and a third patch corresponding to the extractedportion of the third image; may train the AI system to update Model G*to generate second instance classification of features of interest inthe first biological sample, based at least in part on the secondaligned image patches and the labeling of instances of features ofinterest contained in the first set of image patches; and may train theAI system to update Model G to identify instances of features ofinterest in the first biological sample, based at least in part on thefirst patch and the second instance classification of features ofinterest generated by Model G*.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample (including, but notlimited to, at least one of first antigens, first nuclei, first cellwalls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. In some cases, the second imagemay comprise one of highlighting of second features of interest in thefirst biological sample (including, but not limited to, at least one ofsecond antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the second stain that had been applied to thefirst biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain, the second features ofinterest being different from the first features of interest. In someinstances, the third image may comprise one of highlighting of thirdfeatures of interest in the first biological sample (including, but notlimited to, at least one of third antigens, third nuclei, third cellwalls, third cell structures, third antibodies, third normal cells,third abnormal cells, third damaged cells, third cancer cells, thirdtumors, third subcellular structures, third organ structures, or otherfeatures of interest, or the like) by the third stain that had beenapplied to the first biological sample in addition to highlighting ofthe first features of interest by the first stain and highlighting ofthe second features of interest by the second stain, highlighting of thethird features of interest by the third stain in addition to only one ofhighlighting of the first features of interest by the first stain orhighlighting of the second features of interest by the second stain, orhighlighting of the third features of interest by the third stainwithout any of highlighting of the first features of interest by thefirst stain or highlighting of the second features of interest by thesecond stain, and/or the like, the third features of interest beingdifferent from each of the first features of interest and the secondfeatures of interest.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image or a firstlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. Likewise, thesecond image may include, without limitation, one of a second set ofcolor or brightfield images, a second fluorescence image, a second phaseimage, or a second spectral image, and/or the like, where the second setof color or brightfield images may include, but is not limited to, asecond R image, a second G image, and a second B image, and/or the like.The second fluorescence image may include, but is not limited to, atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, and/or the like. The second spectral imagemay include, but is not limited to, at least one of a second Ramanspectroscopy image, a second NIR spectroscopy image, a secondmultispectral image, a second hyperspectral image, or a second fullspectral image, and/or the like. Similarly, the third image may include,without limitation, one of a third set of color or brightfield images, athird fluorescence image, a third phase image, or a third spectralimage, and/or the like, where the third set of color or brightfieldimages may include, but is not limited to, a third R image, a third Gimage, and a third B image, and/or the like. The third fluorescenceimage may include, but is not limited to, at least one of a thirdautofluorescence image or a third labelled fluorescence image having oneor more channels with different excitation or emission characteristics,and/or the like. The third spectral image may include, but is notlimited to, at least one of a third Raman spectroscopy image, a thirdNIR spectroscopy image, a third multispectral image, a thirdhyperspectral image, or a third full spectral image, and/or the like.

In some embodiments, the computing system may receive a fourth image,the fourth image comprising one of a fourth FOV of the first biologicalsample different from the first FOV and the second FOV or a fifth FOV ofa second biological sample, where the second biological sample may bedifferent from the first biological sample; and may identify, usingModel G, second instances of features of interest in the secondbiological sample, based at least in part on the fourth image and basedat least in part on training of Model G using the first patch and thefirst instance classification of features of interest generated by ModelG*. In some cases, the fourth image may comprise the fifth FOV and mayfurther comprise only highlighting of first features of interest in thesecond biological sample by the first stain that had been applied to thesecond biological sample. According to some embodiments, the computingsystem may generate, using Model G, a clinical score, based at least inpart on the identified second instances of features of interest.

In another aspect, the computing system may receive a first image of afirst biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample; and may identify, using afirst AI model (“Model G”) that is generated or updated by a trained AIsystem, first instances of features of interest in the first biologicalsample, based at least in part on the first image and based at least inpart on training of Model G using a first patch and first instanceclassification of features of interest generated by a second model(Model G*) that is generated or updated by the trained AI system byusing first aligned image patches and labeling of instances of featuresof interest contained in the first patch. In some cases, the firstaligned image patches may comprise an extracted portion of first alignedimages, which may comprise a second image and a third image that havebeen aligned. The second image may comprise a second FOV of a secondbiological sample that has been stained with a first stain, the secondbiological sample being different from the first biological sample. Thethird image may comprise a third FOV of the second biological samplethat has been stained with a second stain. The second patch may compriselabeling of instances of features of interest as shown in the extractedportion of the second image of the first aligned images. The first imagemay comprise highlighting of first features of interest in the secondbiological sample by the first stain that had been applied to the secondbiological sample. The second patch may comprise one of highlighting ofsecond features of interest in the second biological sample by thesecond stain that had been applied to the second biological sample inaddition to highlighting of the first features of interest by the firststain or highlighting of the second features of interest by the secondstain without highlighting of the first features of interest by thefirst stain.

In yet another aspect, the computing system may receive first instanceclassification of features of interest (“Ground Truth”) in a firstbiological sample that has been sequentially stained, the Ground Truthhaving been generated by a trained first model (“Model G*”) that hasbeen trained or updated by an AI system, wherein the Ground Truth isgenerated by using first aligned image patches and labeling of instancesof features of interest contained in the first aligned image patches,wherein the first aligned image patches comprise an extracted portion offirst aligned images, wherein the first aligned images comprise a firstimage and a second image that have been aligned, wherein the first imagecomprises a first FOV of the first biological sample that has beenstained with a first stain, wherein the second image comprises a secondFOV of the first biological sample that has been stained with a secondstain, wherein the first aligned image patches comprise labeling ofinstances of features of interest as shown in the extracted portion ofthe first aligned images; and may utilize the AI system to train asecond AI model (“Model G”) to identify instances of features ofinterest in the first biological sample, based at least in part on thefirst instance classification of features of interest generated by ModelG*.

In still another aspect, the computing system may generate ground truthfor developing accurate AI models for biological image interpretation,based at least in part on images of a first biological sample depictingsequential staining of the first biological sample.

In an aspect, the computing system may receive a first image of a firstbiological sample, the first image comprising a first field of view(“FOV”) of the first biological sample that has been stained with afirst stain; may receive a second image of the first biological sample,the second image comprising a second FOV of the first biological samplethat has been stained with at least a second stain; and may align thefirst image with the second image to create first aligned images, byaligning one or more features of interest in the first biological sampleas depicted in the first image with the same one or more features ofinterest in the first biological sample as depicted in the second image.

The computing system may autonomously create first aligned image patchesfrom the first aligned images, by extracting a portion of the firstaligned images, the portion of the first aligned images comprising afirst patch corresponding to the extracted portion of the first imageand a second patch corresponding to the extracted portion of the secondimage; may utilize an AI system to train a first AI model (“Model F”) togenerate a third patch comprising a virtual stain of the first alignedimage patches, based at least in part on the first patch and the secondpatch, the virtual stain simulating staining by at least the secondstain of features of interest in the first biological sample as shown inthe second patch; and may utilize the AI system to train a second model(“Model G*”) to identify or classify first instances of features ofinterest in the first biological sample, based at least in part on thethird patch and based at least in part on results from an externalinstance classification process or a region of interest detectionprocess.

According to some embodiments, the first image may comprise highlightingof first features of interest in the first biological sample (including,but not limited to, at least one of first antigens, first nuclei, firstcell walls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. In some cases, the third patchmay comprise one of highlighting of the first features of interest bythe first stain and highlighting of second features of interest in thefirst biological sample (including, but not limited to, at least one ofsecond antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the virtual stain that simulates the secondstain having been applied to the first biological sample or highlightingof the first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest. In someinstances, the second stain may be one of the same as the first stainbut used to stain the second features of interest or different from thefirst stain.

Merely by way of example, in some cases, the first stain may include,without limitation, at least one of Hematoxylin, Acridine orange,Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystal violet,4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and/or the like.Similarly, the second stain may include, but is not limited to, at leastone of Hematoxylin, Acridine orange, Bismarck brown, Carmine, Coomassieblue, Cresyl violet, Crystal violet, DAPI, Eosin, Ethidium bromideintercalates, Acid fuchsine, Hoechst stain, Iodine, Malachite green,Methyl green, Methylene blue, Neutral red, Nile blue, Nile red, Osmiumtetroxide, Propidium Iodide, Rhodamine, Safranine, PD-L1 stain, DABchromogen, Magenta chromogen, cyanine chromogen, CD3 stain, CD20 stain,CD68 stain, p40 stain, antibody-based stain, or label-free imagingmarker, and/or the like.

In some embodiments, the first image may include, without limitation,one of a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like,where the first set of color or brightfield images may include, but isnot limited to, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. In some instances, the second image may include,without limitation, one of a second set of color or brightfield images,a second fluorescence image, a second phase image, or a second spectralimage, and/or the like, where the second set of color or brightfieldimages may include, but is not limited to, a second R image, a second Gimage, and a second B image, and/or the like. The second fluorescenceimage may include, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike. In some cases, the third patch may include, without limitation,one of a third set of color or brightfield images, a third fluorescenceimage, a third phase image, or a third spectral image, and/or the like,where the third set of color or brightfield images may include, but isnot limited to, a third R image, a third G image, and a third B image,and/or the like. The third fluorescence image may include, but is notlimited to, at least one of a third autofluorescence image or a thirdlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The thirdspectral image may include, but is not limited to, at least one of athird Raman spectroscopy image, a third NIR spectroscopy image, a thirdmultispectral image, a third hyperspectral image, or a third fullspectral image, and/or the like.

According to some embodiments, aligning the first image with the secondimage may be one of performed manually using manual inputs to thecomputing system or performed autonomously, where autonomous alignmentmay comprise alignment using at least one of automated global alignmenttechniques or optical flow alignment techniques.

In some embodiments, training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample may comprise training, using the AI system, Model G* to identifyor classify instances of features of interest in the first biologicalsample, based at least in part on one or more of the first patch, thesecond patch, or the third patch and based at least in part on theresults from the external instance classification process or the regionof interest detection process. In some cases, the external instanceclassification process or the region of interest detection process mayeach include, without limitation, at least one of detection of nuclei inthe first image or the first patch by a nuclei detection method,identification of nuclei in the first image or the first patch by apathologist, detection of features of interest in the first image or thefirst patch by a feature detection method, or identification of featuresof interest in the first image or the first patch by the pathologist,and/or the like.

According to some embodiments, training the AI system to update Model Fmay comprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may include,but is not limited to, one of a mean squared error loss function, a meansquared logarithmic error loss function, a mean absolute error lossfunction, a Huber loss function, or a weighted sum of squareddifferences loss function, and/or the like.

In some embodiments, the first patch may include, without limitation,one of a first set of color or brightfield images, a first fluorescenceimage, a first phase image, or a first spectral image, and/or the like,where the first set of color or brightfield images may include, but isnot limited to, a first red-filtered (“R”) image, a first green-filtered(“G”) image, and a first blue-filtered (“B”) image, and/or the like. Thefirst fluorescence image may include, but is not limited to, at leastone of a first autofluorescence image or a first labelled fluorescenceimage having one or more channels with different excitation or emissioncharacteristics, and/or the like. The first spectral image may include,but is not limited to, at least one of a first Raman spectroscopy image,a first near infrared (“NIR”) spectroscopy image, a first multispectralimage, a first hyperspectral image, or a first full spectral image,and/or the like. In some cases, the second image may include, withoutlimitation, one of a second set of color or brightfield images, a secondfluorescence image, a second phase image, or a second spectral image,and/or the like, where the second set of color or brightfield images mayinclude, but is not limited to, a second R image, a second G image, anda second B image, and/or the like. The second fluorescence image mayinclude, but is not limited to, at least one of a secondautofluorescence image or a second labelled fluorescence image havingone or more channels with different excitation or emissioncharacteristics, and/or the like. The second spectral image may include,but is not limited to, at least one of a second Raman spectroscopyimage, a second NIR spectroscopy image, a second multispectral image, asecond hyperspectral image, or a second full spectral image, and/or thelike. In some instances, the predicted virtually stained image patch mayinclude, without limitation, one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, where the third set of color orbrightfield images may include, but is not limited to, a third R image,a third G image, and a third B image, and/or the like. The thirdfluorescence image may include, but is not limited to, at least one of athird autofluorescence image or a third labelled fluorescence imagehaving one or more channels with different excitation or emissioncharacteristics, and/or the like. The third spectral image may include,but is not limited to, at least one of a third Raman spectroscopy image,a third NIR spectroscopy image, a third multispectral image, a thirdhyperspectral image, or a third full spectral image, and/or the like.

According to some embodiments, operating on the encoded first patch togenerate the color vector may comprise operating on the encoded firstpatch to generate a color vector, using a color vector output and one ofa model architecture, a statistical model-based system, or adeterministic analysis system, and/or the like. In some instances, themodel architecture may comprise at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”), and/or the like. In some cases, the color vector may be one ofa fixed color vector or a learned color vector that is not based on theencoded first patch, and/or the like.

In some embodiments, the image of the virtual stain may include, but isnot limited to, one of a 3-channel image of the virtual stain, aRGB-transform image of the virtual stain, or a logarithmic transformimage of the virtual stain, and/or the like. In some instances,generating the predicted virtually stained image patch may include,without limitation, adding the generated image of the virtual stain tothe received first patch to produce the predicted virtually stainedimage patch. In some cases, determining the first loss value between thepredicted virtually stained image patch and the second patch may beperformed without adding the generated image of the virtual stain to thereceived first patch.

According to some embodiments, the first loss value may include, withoutlimitation, one of a pixel loss value between each pixel in thepredicted virtually stained image patch and a corresponding pixel in thesecond patch or a generative adversarial network (“GAN”) loss valuebetween the predicted virtually stained image patch and the secondpatch, and/or the like. In some instances, the GAN loss value may begenerated based on one of a minimax GAN loss function, a non-saturatingGAN loss function, a least squares GAN loss function, or a WassersteinGAN loss function, and/or the like.

Alternatively, training the AI system to update Model F may comprise:receiving, with the AI system, the first patch; receiving, with the AIsystem, the second patch; generating, with a second model of the AIsystem, an image of the virtual stain; adding the generated image of thevirtual stain to the received first patch to produce a predictedvirtually stained image patch; determining a first loss value betweenthe predicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value.

In some embodiments, the second model may include, without limitation,at least one of a convolutional neural network (“CNN”), a U-Net, anartificial neural network (“ANN”), a residual neural network (“ResNet”),an encode/decode CNN, an encode/decode U-Net, an encode/decode ANN, oran encode/decode ResNet, and/or the like.

According to some embodiments, the computing system may receive a fourthimage, the fourth image comprising one of a fourth FOV of the firstbiological sample different from the first FOV and the second FOV or afifth FOV of a second biological sample, where the second biologicalsample may be different from the first biological sample; and mayidentify, using Model G*, second instances of features of interest inthe second biological sample, based at least in part on the fourth imageand based at least in part on training of Model G* using at least thethird patch comprising the virtual stain of the first aligned imagepatches. According to some embodiments, the computing system maygenerate, using Model G*, a clinical score, based at least in part onthe identified second instances of features of interest.

In some embodiments, the first image may comprise highlighting of firstfeatures of interest in the first biological sample (including, but notlimited to, at least one of first antigens, first nuclei, first cellwalls, first cell structures, first antibodies, first normal cells,first abnormal cells, first damaged cells, first cancer cells, firsttumors, first subcellular structures, first organ structures, or otherfeatures of interest, or the like) by the first stain that had beenapplied to the first biological sample. In some cases, the third patchmay comprise one of highlighting of the first features of interest bythe first stain and highlighting of second features of interest in thefirst biological sample (including, but not limited to, at least one ofsecond antigens, second nuclei, second cell walls, second cellstructures, second antibodies, second normal cells, second abnormalcells, second damaged cells, second cancer cells, second tumors, secondsubcellular structures, second organ structures, or other features ofinterest, or the like) by the virtual stain that simulates the secondstain having been applied to the first biological sample or highlightingof the first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest. In someinstances, the second stain may be one of the same as the first stainbut used to stain the second features of interest or different from thefirst stain.

According to some embodiments, the first image may include, withoutlimitation, one of a first set of color or brightfield images, a firstfluorescence image, a first phase image, or a first spectral image,and/or the like, where the first set of color or brightfield images mayinclude, but is not limited to, a first red-filtered (“R”) image, afirst green-filtered (“G”) image, and a first blue-filtered (“B”) image,and/or the like. The first fluorescence image may include, but is notlimited to, at least one of a first autofluorescence image or a firstlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The firstspectral image may include, but is not limited to, at least one of afirst Raman spectroscopy image, a first near infrared (“NIR”)spectroscopy image, a first multispectral image, a first hyperspectralimage, or a first full spectral image, and/or the like. In someinstances, the second image may include, without limitation, one of asecond set of color or brightfield images, a second fluorescence image,a second phase image, or a second spectral image, and/or the like, wherethe second set of color or brightfield images may include, but is notlimited to, a second R image, a second G image, and a second B image,and/or the like. The second fluorescence image may include, but is notlimited to, at least one of a second autofluorescence image or a secondlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, and/or the like. The secondspectral image may include, but is not limited to, at least one of asecond Raman spectroscopy image, a second NIR spectroscopy image, asecond multispectral image, a second hyperspectral image, or a secondfull spectral image, and/or the like. In some cases, the third patch mayinclude, without limitation, one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, and/or the like, where the third set of color orbrightfield images may include, but is not limited to, a third R image,a third G image, and a third B image, and/or the like. The thirdfluorescence image may include, but is not limited to, at least one of athird autofluorescence image or a third labelled fluorescence imagehaving one or more channels with different excitation or emissioncharacteristics, and/or the like. The third spectral image may include,but is not limited to, at least one of a third Raman spectroscopy image,a third NIR spectroscopy image, a third multispectral image, a thirdhyperspectral image, or a third full spectral image, and/or the like.

In some embodiments, training the AI system to update Model F maycomprise: receiving, with an encoder, the first patch; receiving thesecond patch; encoding, with the encoder, the received first patch;decoding, with the decoder, the encoded first patch; generating anintensity map based on the decoded first patch; simultaneously operatingon the encoded first patch to generate a color vector; combining thegenerated intensity map with the generated color vector to generate animage of the virtual stain; adding the generated image of the virtualstain to the received first patch to produce a predicted virtuallystained image patch; determining a first loss value between thepredicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value. In some instances, the loss function may include,but is not limited to, one of a mean squared error loss function, a meansquared logarithmic error loss function, a mean absolute error lossfunction, a Huber loss function, or a weighted sum of squareddifferences loss function, and/or the like.

According to some embodiments, the first loss value may include, withoutlimitation, one of a pixel loss value between each pixel in thepredicted virtually stained image patch and a corresponding pixel in thesecond patch or a generative adversarial network (“GAN”) loss valuebetween the predicted virtually stained image patch and the secondpatch, and/or the like. In some cases, the GAN loss value may begenerated based on one of a minimax GAN loss function, a non-saturatingGAN loss function, a least squares GAN loss function, or a WassersteinGAN loss function, and/or the like.

In yet another aspect, the computing system may receive a first image ofa first biological sample, the first image comprising a first field ofview (“FOV”) of the first biological sample; and may identify, using afirst model (“Model G*”) that is generated or updated by a trained AIsystem, first instances of features of interest in the first biologicalsample, based at least in part on the first image and based at least inpart on training of Model G* using at least a first patch comprising avirtual stain of first aligned image patches, the first patch beinggenerated by a second AI model (Model F) that is generated or updated bythe trained AI system by using a second patch. In some cases, the firstaligned image patches may comprise an extracted portion of first alignedimages. The first aligned images may comprise a second image and a thirdimage that have been aligned. The second image may comprise a second FOVof a second biological sample that is different from the firstbiological sample that has been stained with a first stain. The secondpatch may comprise the extracted portion of the second image. The thirdimage may comprise a third FOV of the second biological sample that hasbeen stained with at least a second stain.

These and other functions of the system 1100 (and its components) aredescribed in greater detail above with respect to FIGS. 3-9 .

Reference is now made to FIG. 23 , which is a flowchart of a method oftraining a virtual stainer ML model, in accordance with variousembodiments.

At 2302, a sample of tissue of a subject stained with a removable stainis provided. Exemplary tissue samples are, for example, as describedherein.

Exemplary removable stains include Hematoxyline and Eosin (H&E), and/ora non-labelled scan. Examples of non-labelled scans include: ramanspectroscopy, autofluorescence, darkfield, and pure contrast. Otherexamples of removable stains and/or non-labelled scans are, for example,as described herein.

At 2304, an image is captured depicting the sample of tissue stainedwith the removable stain. For example, a whole slide image in color andhigh resolution. Exemplary approaches for capturing images of tissuesamples are, for example, as described herein.

At 2306, the removable stain is cleared from the tissue sample to obtaina cleared tissue sample. The removable stain is cleared after the imageof the tissue sample stained with the removable stain is captured.

For tissues stained with H&E, the H&E stain may be removed, for example,by incubating the stained tissue sample for about 30 minutes in about96% ethanol to dissolve eosin. The tissue sample may be subject to about0.3 M H2SO4 for about 3 minutes to remove the hematoxylin. Otherapproaches may be used, for example, leaving the tissue sample in about70% acidified ethanol for a long period of time to remove the H&E stain.

For tissue samples “stained” with non-labelled scans, the appliedelectromagnetic energy may be terminated. For example, the ramanspectroscopy, the fluorescence, and darkfield, the respectiveelectromagnetic energies are stopped.

It is noted that permanent stains cannot be removed from the tissuesample. Even if the permanent stain could somehow be removed, theremoval process damages the cleared tissue sample such that the clearedtissue sample cannot be analyzed to obtain useful pathological and/orclinical information. Special stains that are removable are referred toherein as removable stains. It is noted that the majority of specialstains are permanent, and are referred to herein as permanent stains.

It is further noted that multiple removable stains may be applied andremoved, one after the other, with an optional final permanent stainapplied. This may enable creating multiple pairs of images using thesame tissue, using pairs of removable stains. The method described withreference to FIG. 23 may be implemented using such approach, when in afirst iteration a first record includes an image of first removablestain and a ground truth of a second removable stain, in a seconditeration a second record includes an image of the second removablestain and a ground truth of a third removable stain, and in a thirditeration a third record includes an image of the third removable stainand of a permanent stain. It is noted that the previous example is notnecessarily limiting, and that different records may be created byrepeatedly clearing removable stains.

At 2308, a permanent stain is applied to the cleared tissue sample. Thepermanent stain cannot be removed, for example, the permanent staincannot be removed and replaced with another permanent stain.

Examples of permanent stains include a non-H&E stain, a special stain,and/or an immunohistochemistry (IHC) marker stain. Examples of permanentstains (and/or special stains) include: the sequential stain describedherein, PDL1, HER2 immunohistochemistry (IHC) stain, Masson’s trichrome,Jones Silver H&E, Periodic Acid - Schiff (PAS), and the like.

At 2310, an image is captured depicting the sample of tissue (i.e., thecleared tissue sample) of the subject stained with the permanent stain.The tissue sample depicted in the image with permanent stain is the sametissue sample depicted in the image with removable stain described withreference to 2304.

Using the same tissue sample for the training input and for the groundtruth improves performance of the trained visual stainer machinelearning model. Using the same tissue sample helps the visual stainer MLmodel to focus on learning the difference in staining between the twoimages, rather than also having to cope with structural differences. Inparticular, aligning the two images of the same tissue sample accordingto structural features further improves performance of the visualstainer ML model, by reducing and/or removing structural differencesbetween the two images that arose due to physical processing of theimages during the staining process. Such structures differences areundesired artifacts that may reduce performance of the visual stainer MLmodel and/or reduce effectiveness of the training, if not removed.Alignment of the images is now described with reference to 2312.

At 2312, the image of the tissue stained with the removable stain (e.g.,as described with reference to 2304) may be aligned with the imagestained with the permanent stain (e.g., as described with reference to2310). The alignment process is designed to reduce and/or removestructural differences between the two images that arose, for example,due to physical processing of the images during the staining process.

An exemplary alignment process is now described. Multiple biologicalfeatures depicted in the image of the tissue stained with the removablestain are identified. Multiple biological features depicted in the imageof the tissue stained with the permanent stain are identified. At leastsome of the biological features of the image of the tissue stained withthe removable stain correspond to the biological features of the imageof the tissue stained with the permanent stain. The correspondence ofbiological features is based on the fact that it is the same tissuesample, with different stains, that is depicted in both images. Anoptical flow process and/or non-rigid registration process is applied toone of the images to compute a respective aligned image. The alignmentprocess may be applied in two steps. In a first step, the non-rigidregistration is performed to obtain an initial rough alignment. In asecond step, the optical flow process is performed using the initialrough alignment, to obtain a fine alignment between the two images.Alternatively, one of the optical flow process and the non-rigidregistration process is applied, in a single step. The optical flowprocess and/or non-rigid registration process aligns pixels locations ofthe aligned image to corresponding pixel locations of the other imageusing optical flow computed between the biological features of thealigned image and the biological features of the other image. Thealigned image may serve as the ground truth. Additional exemplarydetails of the optical flow process and/or non-rigid registrationprocess are, for example, as described herein. Other exemplaryapproaches for alignment of the images are, for example, as describedherein.

At 2314, a record is created from the image of tissue stained with theremovable stain (e.g., as described with reference to 2304) and from theimage of tissue stained with the permanent stain (e.g., as describedwith reference to 2310).

The record includes a first image representing a training input image,i.e., an example of an image that will be fed into the trained virtualstainer ML model. The record further includes a second imagerepresenting a ground truth image, i.e., an example of what a syntheticimage looks like, where the synthetic image will be created by thetrained virtual stainer ML model in response to being fed an inputimage. It is noted that in the training dataset, the ground truth imageis real image of a real tissue sample stained with the selected stain,whereas during inference, the virtual stainer ML model learns tosynthesize an synthetic image that looks like the real image. Thevirtual stainer ML model is trained on the record, to learn tosynthesize a synthetic image that looks like the ground truth image inresponse to being fed an input image that looks like the training inputimage.

Optionally, the image of tissue stained with the removable stainrepresents an example training input, and the image of the tissuestained with the permanent stain represents a ground truth.

Alternatively, the image of tissue stained with the permanent stainrepresents an example training input, and the image of the tissuestained with the removable stain represents a ground truth.

Reference is now made to FIG. 25 , which is a flowchart of an exemplaryprocess for generating a first image denoting a training input image,and a ground truth second image of a record of a multi-record trainingdataset for training a virtual stainer, in accordance with variousembodiments. The first image and the ground truth second image are ofthe same tissue sample, stained with different stains.

At 2502, a slide of a tissue sample of a subject stained with aremovable stain, for example, H&E, is provided. The H&E stained samplemay be created, for example, using a standard protocol Dako CoverStainermachine from Agilent®, or other devices.

At 2504, the H&E stained tissue sample is scanned, for example, by adigital slide scanner, such as a Phillips Ultrafast machine.

At 2506, the coverslip is removed from the H&E stained tissue sample.The removable stain is cleared from the tissue sample, to obtain acleared tissue sample. For example, by incubating the stained tissuesample for about 30 minutes in about 96% ethanol to dissolve eosin. Thetissue sample may be subject to about 0.3 M H2SO4 for about 3 minutes toremove the hematoxylin. Other approaches may be used, for example,leaving the tissue sample in about 70% acidified ethanol for a longperiod of time.

At 2508, a permanent stain, optionally a special stain, is applied tothe cleared tissue sample. The special stain is applied to the sametissue sample that was previously stained with the removable stain. Thespecial stain may be applied, for example, using the Artisan Pro withspecial stain and/or kit available from Agilent® or using other devices.

At 2510, a second image of the tissue sample stained with the permanentstain is captured, optionally by the same digital slide scanner used toobtain the first image (e.g., as described with reference to 2504). Thesecond image serves as the ground truth, as described herein.

Referring now back to FIG. 23 , at 2316, an imaging multi-recordtraining dataset is created. Features described with reference to2302-2314 create a single record. The features described with referenceto 2302-2314 may be iterated to create multiple records, for example,each iteration is for another tissue sample of another subject.

Multiple records may be included in a single training dataset.Optionally, the records included in the training dataset are for a samepair of removable and permanent stain, optionally of a same type oftissue from different subjects. For example, kidney tissues obtained viabiopsies from different subjects, where each record of the kidney tissueincludes an image depicting the H&E stain and another image depictingthe Masson’s trichrome stain.

Optionally, multiple training datasets are created. Each trainingdataset is created as described herein for creating the single trainingdataset, with a variation of the records between the different trainingdataset. The variation may be in the stains used in the images. Thevariation may be in the permanent stain used in the ground truth images,where the removable stain used in the input training image may be thesame across all training dataset. The type of tissue may be similarbetween training dataset, or different between training dataset. Forexample, all training datasets are of kidney tissue, and all traininginput images depict the H&E stain. The permanent stain may vary betweentraining dataset, for example, the Masson’s trichrome stain in a firsttraining dataset, the Jones Silver stain in a second training dataset,and the PAS stain in a third training dataset.

At 2318, a virtual stainer machine learning model is trained on theimaging multi-record training dataset.

Optionally, when multiple imaging multi-record training datasets arecreated, multiple virtual stainer ML models are trained. Each respectivevirtual stainer ML model is trained on another imaging multi-recordtraining dataset.

Alternatively, a single main virtual stainer ML model is trained on themultiple imaging multi-record training dataset, and/or on a single maintraining dataset that includes records of different images withdifferent stain combinations. In such implementation where the virtualstainer ML model is trained on records that include images withdifferent stains (e.g., images depicting different combinations ofstains), each record may further include an indication of the staindepicted in the ground truth image and/or an indication of the staindepicted in the training input image. The indication, when provided asinput into the virtual stainer ML model in combination with an inputimage, directs the virtual stainer ML model as to the desired stain touse in the generated output image.

Optionally, the virtual stainer machine learning model is implemented asa generative adversarial network (GAN) comprising a generator networkand discriminator network. The generative network is trained forgenerating the virtual image using a loss function computed based ondifferences between pixels of a real ground truth image obtained from arecord of the training dataset, and pixels of an outcome of a virtualimage created by the generative network in response to an input of aninput training image of the record of the training dataset. The lossfunction may be alternatively or additionally computed according to anability of the discriminator network to differentiate between theoutcome of the virtual image created by the generative network and thereal image.

Additional exemplary implementations of the virtual stainer ML model,and/or additional exemplary details of training the virtual stainer MLmodel are described, for example, with reference to FIG. 26 and/orwithin the disclosure.

Reference is now made to FIG. 26 , which is a dataflow diagram depictingexemplary dataflow for training a generative adversarial network (GAN)implementation of the virtual stainer, in accordance with variousembodiments. The GAN includes a generative network and a discriminatornetwork. The GAN learns to generate virtual images stained with acertain stain that are highly reminiscent of real images stained withreal stains, in response to an input of another image of the same tissuestained with another stain.

A first image 2602 of a record of the multi-record training dataset isprovided. The first image is stained using a first stain, for example, aremovable stain such as H&E.

A second image 2604, denoting ground truth, of the record of themulti-record training dataset is provided. The second image is stainedusing a second stain, for example, a permanent stain such as a specialstain. It is noted that other records may be used, for example the firstimage is of the permanent stain and the second image denoting groundtruth is of the removable stain.

The first image and the second image may be aligned, optionally to pixellevel precision, for example, using a first rough alignment followed bya finer optical flow based alignment, for example, as described herein.

A generator network 2606 of the GAN learns to generate synthetic virtualimages depicting the second stain 2608, in response to an input of thefirst image depicting the first stain (e.g., 2602).

An optimizer 2610 optimizes generator network 2606 according to a lossfunction 2612 computed based on differences between pixels of the secondimage 2614 generated by generator network 2606 in response to input offirst image 2602, and the real second image 2604. Loss function 2612 maybe based on the difference between pixel to pixel differences betweenpair of real images 2604 and pair of first real image 2602 and secondsynthesized image 2608 generated generator network 2606 in response toinput of first real image 2602.

Optimizer 2610 further optimizes generator network 2606 based on anability of the discriminator network 2616 to differentiate between theoutcome of the virtual second image 2608 created by generative network2606 (in response to an input of first real image 2602) and the realsecond image 2604. Discriminator network 2616 may generate output 2618which is fed into optimizer 2610 for optimizing generator network 2606.Output 2618 may be a P value indicative of output based on input is real(i.e., real pair) and/or P for output based on input is real (i.e.,generative). The GAN may generate the P value for subsets of the image(e.g., tile).

Once training is completed, generator network 2606 is provided forgenerating synthetic virtual images depicting the second stain inresponse to an input of a first image depicting the first stain. Thesynthetic virtual images are of high enough quality to “fool”discriminator network 2616 such that discriminator network 2616 cannottell if the synthetic virtual image is a real image or a synthetic imagecreated by generator network 2606. The threshold of the ability to“fool” the discriminator network may be set to a level corresponding anability to “fool” a human observer, such that the human observer cannottell if the synthetic virtual image created by generator network B06 isreal or synthetic.

Referring now back to FIG. 23 , at 2320, the trained virtual stainer MLmodel(s) is provided. The virtual stainer ML model may be provided, forexample, to a remote client terminal for local use by the clientterminal, to a central server for providing centralized services tomultiple clients, to another process, and/or stored on a data storagedevice.

Reference is now made to FIG. 24 , which is a flowchart of a method ofinference using a trained virtual stainer ML model, in accordance withvarious embodiments.

At 2402, a target input image, for being fed into a trained virtualstainer ML model, is obtained. For example, the target input image isobtained from one or more of: automatically from a PACS server (e.g., inresponse to being loaded to the PACS), automatically obtained from atissue scanner, manually loaded by a user from a data storage device,received from a client terminal over a network, and the like.

The target input image is stained with a permanent stain, or with aremovable stain.

At 2404, optionally, when there are multiple trained virtual stainer MLmodels available (trained on different imaging multi-record trainingdatasets having different permanent stains depicted in the ground truthimage, for example, as described herein) the trained virtual stainer MLmodel may be selected. The trained virtual stainer ML model may beselected according to a desired target stain to be depicted in asynthetic image generated by the selected virtual stainer ML model.Alternatively or additionally, the trained virtual stainer ML model maybe selected according to the stain depicted in the input image, i.e.,that the images of the records used to train the selected virtualstainer depicted the same type of stain. Alternatively or additionally,the trained virtual stainer ML model may be selected according to thetype of tissue depicted in the input image, i.e., that the images of therecords used to train the selected virtual stainer depicted the sametype of tissue.

One or more virtual stainer ML models may be selected.

One or more target stains (to be depicted in the synthetic image) may beselected.

The virtual stainer ML model and/or the target stain (to be depicted inthe synthetic image) may be selected using different approaches. Someexemplary approaches are now described:

In one approach, the user manually performs the selection. For example,the user may click on an icon indicating the desired target stain to beobtained in the synthesized image, and/or on an icon indicating the typeof tissue depicted in the input image. The relevant virtual stainer MLmodel may be selected automatically in response to the manual userinput.

In another approach, the selection is automatically performed. Forexample, the target input image is analyzed to identify a type of thesample of tissue (from multiple defined types, such as organ, forexample, kidney, liver, blood, and the like). The desired target stainto be obtained in the synthesized image may be determined according tothe identified type of tissue. For example, certain special stains arecommonly used for certain tissue types. Those special stains areautomatically selected in response to identifying the type of tissue.For example, when liver tissue is identified in the target input image,Masson’s trichrome, Jones Silver H&E, and PAS special stains areautomatically selected.

The identification of the tissue type and/or selection of the specialstain may be performed, for example, by a set of rules, an analysisprocess that analyzes features of the input image, another trained MLmodel, manually, and the like. For example, another ML model may betrained on a training dataset of slides labelled with type of tissue,for automatically detecting the tissue type on an input image. Inanother example, a set of rules may map the detected tissue type to oneor more special stains. In yet another example, yet another ML model maybe trained on a training dataset that is created by learning theworkflow of real pathologists, such as which special stain thepathologist selects for each corresponding image.

At 2406, the target input image of the sample of tissue is fed into theselected virtual stainer ML model(s). When multiple models are selected,the same target input image is fed into each one of the virtual stainerML model(s). When there is a main virtual stainer ML model that iscapable of generating synthetic images with different stains, the targetinput image may be fed into the main virtual stainer ML model incombination with an indication of the desired target stain(s). Forexample, the indication of the desired target stain(s) may be metadata,and/or a code indicating which one of the possible stains is beingselected.

At 2408, a synthetic image is obtained as an outcome of the virtualstainer ML model. The synthetic image depicts the same tissue as in thetarget input image, stained with the target stain.

Multiple synthetic images with different stains, and depicting the sametissue as in the target input image may be obtained. The multiplesynthetic images may be obtained from different virtual stainer MLmodels, and/or from the same virtual stainer ML model which may be fedthe indication(s) of which stains to provide.

At 2410, the synthetic image(s) are provided. For example, the syntheticimage(s) may be presented on a display, for example, within a userinterface (e.g., GUI) designed to enable the user to view, zoom, and/ormark the synthetic image. The synthetic image(s) may be presentedalongside the target input image. The synthetic image(s) may beforwarded to another device, such as over a network. The syntheticimage(s) may be saved, for example, in a PACS, in an HER, and/or otherimplementations. The synthetic image(s) may be forwarded to anotherprocess, for example, for automated analysis, such as for automateddiagnosis, and the like.

Various embodiments and/or aspects as delineated hereinabove and asclaimed in the claims section below find experimental and/or calculatedsupport in the following examples.

EXAMPLES

Reference is now made to the following examples, which together with theabove descriptions illustrate some embodiments of the invention in a nonlimiting fashion.

Reference is now made to FIG. 27 , which includes example images of realand virtual slides, as part of an experiment performed by Inventors, inaccordance with various embodiments. A virtual stainer ML model wascreated and trained by Inventors to be fed an input of an image stainedwith H&E, and create a synthetic virtual image of the same tissue samplestained with Masson’s trichrome (which may be used for cases ofcirrhotic liver disease), as described herein. An input image 2702depicting tissue with cirrhotic liver disease, stained with theremovable stain H&E, was fed into the virtual stainer ML model. Image2704 depicts the output of the virtual stainer ML model. Synthetic image2704 is of the same tissue sample as input image 2702, stained with avirtual Masson’s trichrome stain. Real image 2706 was created byremoving the H&E stain from the tissue sample (depicted in image 2702),and applying a real Masson’s trichrome stain to the cleared tissuesample. It is visually apparent that synthetic image 2704 and real image2706 cannot be differentiated by an observer, indicating ability of thegenerator network to accurately create synthetic virtual images, asdescribed herein.

Reference is now made to FIG. 28 , which includes example images ofadditional real and virtual slides, as part of another experimentperformed by Inventors, in accordance with various embodiments. Avirtual stainer ML model was created and trained by Inventors to be fedan input of an image stained with H&E, and create a synthetic virtualimage of the same tissue sample stained with HER2 IHC (which may be usedfor cases of breast carcinoma), as described herein. An input image 2802depicting tissue with breast carcinoma, stained with the removable stainH&E, was fed into the virtual stainer ML model. Image 2804 depicts theoutput of the virtual stainer ML model. Synthetic image 2804 is of thesame tissue sample as input image 2802, stained with a virtual HER2 IHCstain. Real image 2806 was created by removing the H&E stain from thetissue sample (depicted in image 2802), and applying a real HER2 IHCstain to the cleared tissue sample. It is visually apparent thatsynthetic image 2804 and real image 2806 cannot be differentiated by anobserver, indicating ability of the generator network to accuratelycreate synthetic virtual images, as described herein.

Reference is now made to FIG. 29 , which includes example images of yetadditional real and virtual slides, as part of yet another experimentperformed by Inventors, in accordance with various embodiments. Avirtual stainer ML model was created and trained by Inventors to be fedan input of an image stained with IHC (i.e., a permanent and/or specialstain), and create a synthetic virtual image of the same tissue samplestained with H&E (i.e., a removable stain), as described herein. Aninput image 2902 depicting tissue with breast carcinoma, stained withIHC, was fed into the virtual stainer ML model. Image 2904 depicts theoutput of the virtual stainer ML model. Synthetic image 2904 is of thesame tissue sample as input image 2902, stained with H&E.

It is noted that a real image corresponding to synthetic image 2904cannot be created once IHC has been applied, since even if one were toattempt to remove the IHC stain from input image 2902, such attemptwould destroy the tissue sample, rendering it useless for pathologicalexamination. However, by first applying a real H&E stain to the tissuesample to get an image similar to 2904, removing the H&E stain, and thenapplying the IHC stain to obtain another image similar to 2902, it ispossible to obtain a pair of images depicting the same tissue sample,one image depicting tissue stained with IHC and a second image depictingtissue stained with H&E.

Reference is now made to FIG. 30 , which includes example images of yetadditional real and virtual slides, as part of yet another experimentperformed by Inventors, in accordance with various embodiments. Threevirtual stainer ML models were created and trained by Inventors. Eachvirtual stainer ML model was trained to be fed the same input imagestained with H&E, and create a respective synthetic virtual imagestained with a respective stain, as described herein. An input image3002 depicting kidney tissue, stained with IHC, was fed into each one ofthe three virtual stainer ML models. Images 3004, 3006, and 3008 depictsthe output of the first, second, and third virtual stainer models.Synthetic image 3004 is of the same tissue sample as input image 3002,stained with Masson’s trichrome. Synthetic image 3006 is of the sametissue sample as input image 3002, stained with Jones Silver H&E.Synthetic image 3008 is of the same tissue sample as input image 3002,stained with Periodic Acid - Schiff (PAS). Image 3014 is of real kidneytissue stained with Masson’s trichrome, for comparison with syntheticimage 3004. Image 3016 is of real kidney tissue stained with JonesSilver H&E, for comparison with synthetic image 3006. Image 3018 is ofreal kidney tissue stained with PAS, for comparison with synthetic image3008. It is noted that real images of the same kidney tissue cannot beprepared for multiple different permanent stains, since after onepermanent stain is applied, the permanent stain cannot be cleared inorder to apply another permanent stain. It is visually apparent thatsynthetic images 3004 3006 and 3008 and corresponding real images 30143016 and 3018 cannot be differentiated by an observer, indicatingability of the generator network to accurately create synthetic virtualimages, as described herein.

Additional exemplary embodiments are now described.

According to an aspect of some embodiments of the present inventionthere is provided a method for detecting multiple target molecules in abiological sample comprising cells, the method comprising: contactingthe biological sample with one or more first reagents which generate adetectable signal in cells comprising a first target molecule;contacting the biological sample with one or more second reagents whichare capable of generating a detectable signal in cells comprising asecond target molecule under conditions in which the one or more secondreagents do not generate a detectable signal; detecting the signalgenerated by the one or more first reagents; creating conditions inwhich the one or more second reagents generate a signal in cellscomprising the second molecule; and detecting the signal generated bythe one or more second reagents.

Optionally, further comprising: obtaining a first digital image of thesignal generated by the one or more first reagents; obtaining a seconddigital image of the signal generated by the one or more first reagentsand the signal generated by the one or more second reagents; and copyinga mask of the second digital image to the first digital image.

Optionally, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

Optionally, the target molecules are indicative of at least one ofnormal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

Optionally, the target molecules are selected from the group consistingof nuclear proteins, cytoplasmic proteins, membrane proteins, nuclearantigens, cytoplasmic antigens, membrane antigens and nucleic acids.

Optionally, the target molecules are nucleic acids.

Optionally, the target molecules are polypeptides.

Optionally, the first reagents comprise a first primary antibody againsta first target polypeptide, an HRP coupled polymer that binds to thefirst primary antibody and a chromogen which is an HRP substrate.

Optionally, the second reagents comprise a second primary antibodyagainst a second target polypeptide, an HRP coupled polymer that bindsto the second primary antibody, a fluorescein-coupled long singlechained polymer, an antibody against FITC that binds to thefluorescein-coupled long single chained polymer and is coupled to HRPand a chromogen which is an HRP substrate.

Optionally, further comprising using a digital image of the signalsdetected in the biological sample to train a neural network.

Optionally, a neural network is trained using the method

According to an aspect of some embodiments of the present inventionthere is provided a method for generating cell-level annotations from abiological sample comprising cells comprising:

-   a) exposing the biological sample to a first ligand that recognizes    a first antigen thereby forming a first ligand antigen complex;-   b) exposing the first ligand antigen complex to a first labeling    reagent binding to the first ligand, the first labeling reagent    forming a first detectable reagent, whereby the first detectable    reagent is precipitated around the first antigen and visible in    brightfield;-   c) exposing the biological sample to a second ligand that recognizes    a second antigen thereby forming a second ligand antigen complex;-   d) exposing the second ligand antigen complex to a second labeling    reagent binding to the second ligand, the second labeling reagent    comprising a substrate not visible in brightfield, whereby the    substrate is precipitated around the second antigen;-   e) obtaining a first image of the biological sample in brightfield    to visualize the first chromogen precipitated in the biological    sample;-   f) exposing the biological sample to a third labeling reagent that    recognizes the substrate, thereby forming a third ligand antigen    complex, the third labeling reagent forming a second detectable    reagent, whereby the second detectable reagent is precipitated    around the second antigen;-   g) obtaining a second image of the biological sample in brightfield    with the second detectable reagent precipitated in the biological    sample;-   h) creating a mask from the second image; and-   i) applying the mask to the first image so as to obtain an image of    the biological sample annotated with the second antigen.

Optionally, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

Optionally, the target molecules are indicative of at least one ofnormal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

Optionally, the method further comprises, prior to step a), applying atarget retrieval buffer and protein blocking solution to the biologicalsample, whereby the first and second antigens are exposed for thesubsequent steps and endogenous peroxidases are inactivated.

Optionally, the first and third labeling reagents comprise an enzymethat acts on a detectable reagent substrate to form the first and seconddetectable reagents, respectively.

Optionally, the first and second antigens are non-nuclear proteins.

Optionally, the method further comprises, following step b), denaturingthe first ligands to retrieve the first antigens available.

Optionally, the method further comprises, following step d),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

Optionally, the method further comprises, following step e),

-   i) removing mounting medium from the slide, and-   ii) rehydrating the biological sample.

Optionally, the method further comprises, following step f),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

Optionally, the first ligand comprises an anti- lymphocyte-specificantigen antibody (“primary antibody”), the first labeling reagentcomprises a horseradish peroxidase (HRP)-coupled polymer capable ofbinding to the primary antibody, the first detectable reagent comprisesa chromogen comprising an HRP substrate comprising HRP Magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB), the second antigencomprises PD-L1, the second ligand comprises anti-PD-L1 antibodies, thesecond labeling reagent comprises an HRP-coupled polymer capable ofbinding to anti-PD-L1 antibodies, the substrate comprisesfluorescein-coupled long single-chained polymer (fer-4-flu linker),third labeling reagent comprises an anti-FITC antibody coupled to HRP,and the second detectable reagent comprises a chromogen comprising HRPMagenta or DAB.

Optionally, the first antigen comprises PD-L1. the first ligandcomprises anti-PD-L1 antibodies, the first labeling reagent comprises anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, thesubstrate comprises fluorescein-coupled long single-chained polymer(fer-4-flu linker), the second ligand comprises ananti-lymphocyte-specific antigen antibody (“primary antibody”), thesecond labeling reagent comprises a horseradish peroxidase (HRP)-coupledpolymer capable of binding to the primary antibody, the seconddetectable reagent comprises a chromogen comprising an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), third labeling reagent comprises an anti-FITC antibody coupled toHRP, and the second detectable reagent comprises a chromogen comprisingHRP Magenta or DAB.

Optionally, the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises a chromogen comprisingan HRP substrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), the second antigen comprises p40, the secondligand comprises anti-p40 antibodies, the second labeling reagentcomprises an HRP-coupled polymer capable of binding to anti-p40antibodies, the substrate comprises fluorescein-coupled longsingle-chained polymer (fer-4-flu linker), third labeling reagent ananti-FITC antibody coupled to HRP, and the second detectable reagentcomprises a chromogen comprising HRP Magenta or DAB.

Optionally, the first antigen comprises p40, the first ligand comprisesanti-p40 antibodies, the first labeling reagent comprises an HRP-coupledpolymer capable of binding to anti-p40 antibodies, the substratecomprises fluorescein-coupled long single-chained polymer (fer-4-flulinker), the second labeling reagent comprises a horseradish peroxidase(HRP)-coupled polymer capable of binding to the primary antibody, thesecond detectable reagent comprises a chromogen comprising an HRPsubstrate comprising HRP Magenta or 3,3′-diaminobenzidinetetrahydrochloride (DAB), third labeling reagent an anti-FITC antibodycoupled to HRP, and the second detectable reagent comprises a chromogencomprising HRP Magenta or DAB.

Optionally, a counterstaining agent is hematoxylin.

Optionally, the method further comprises using a digital image of thesignals detected in the biological sample to train a neural network.

Optionally, a neural network trained using the method.

Optionally, an annotated image obtained by the method.

According to an aspect of some embodiments of the present inventionthere is provided a method for detecting multiple target molecules in abiological sample comprising cells, the method comprising: contactingthe biological sample with one or more first reagents which generate adetectable signal in cells comprising a first target molecule;contacting the biological sample with one or more second reagents whichgenerate a detectable signal in cells comprising a second targetmolecule, wherein the detectable signal generated by the one or moresecond reagents is removable; detecting the signal generated by the oneor more first reagents and the signal generated by the one or moresecond reagents; creating conditions in which the signal generated bythe one or more second reagents is removed; and detecting the signalgenerated by the one or more first reagents.

Optionally, the method further comprises: obtaining a first digitalimage of the signal generated by the one or more first reagents and thesignal generated by the one or more second reagents; obtaining a seconddigital image of the signal generated by the one or more first reagents;and copying a mask of the first digital image to the second digitalimage.

Optionally, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

Optionally, the target molecules are indicative of at least one ofnormal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

Optionally, the target molecules are selected from the group consistingof nuclear proteins, cytoplasmic proteins, membrane proteins, nuclearantigens, cytoplasmic antigens, membrane antigens and nucleic acids.

Optionally, the target molecules are nucleic acids.

Optionally, the target molecules are polypeptides.

Optionally, the first reagents comprise a first primary antibody againsta first target polypeptide, an HRP coupled polymer that binds to thefirst primary antibody and a chromogen which is an HRP substrate.

Optionally, the second reagents comprise a second primary antibodyagainst a second target polypeptide, an HRP coupled polymer that bindsto the second primary antibody, and amino ethyl carbazole.

Optionally, the method further comprises using a digital image of thesignals detected in the biological sample to train a neural network.

Optionally, a neural network trained using the method.

According to an aspect of some embodiments of the present inventionthere is provided a method for generating cell-level annotations from abiological sample comprising cells, the method comprising:

-   a) exposing the biological sample to a first ligand that recognizes    a first antigen thereby forming a first ligand antigen complex;-   b) exposing the first ligand antigen complex to a first labeling    reagent binding to the first ligand, the first labeling reagent    forming a first detectable reagent, whereby the first detectable    reagent is precipitated around the first antigen;-   c) exposing the biological sample to a second ligand that recognizes    a second antigen thereby forming a second ligand antigen complex;-   d) exposing the second ligand antigen complex to a second labeling    reagent binding to the second ligand, the second labeling reagent    forming a second detectable reagent, whereby the second detectable    reagent is precipitated around the second antigen;-   e) obtaining a first image of the biological sample with the first    and second detectable reagents precipitated in the biological    sample;-   f) incubating the tissue sample with an agent which dissolves the    second detectable reagent;-   g) obtaining a second image of the biological sample with the first    detectable reagent precipitated in the biological sample;-   h) creating a mask from the first image; and-   i) applying the mask to the second image so as to obtain an    annotated image of the biological sample with the second antigen.

Optionally, the first biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample, wherein theobjects of interest comprise at least one of normal cells, abnormalcells, damaged cells, cancer cells, tumors, subcellular structures, ororgan structures.

Optionally, the method further comprises, prior to step a), applying atarget retrieval buffer and protein blocking solution to the biologicalsample, whereby the first and second antigens are exposed for thesubsequent steps and endogenous peroxidases are inactivated.

Optionally, the first and second labeling reagents comprise an enzymethat acts on a detectable reagent substrate to form the first and seconddetectable reagents, respectively.

Optionally, the first and second antigens are non-nuclear proteins.

Optionally, further comprising, following step b), denaturing the firstligands to retrieve the first antigens available.

Optionally, the method further comprises, following step d),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

Optionally, further comprising, following step e),

-   i) removing mounting medium from the slide, and-   ii) rehydrating the biological sample.

Optionally, the method further comprises, following step f), dehydratingand mounting the sample on a slide.

Optionally, the first antigen comprises a lymphocyte-specific antigen,the first ligand comprises an anti-lymphocyte-specific antigen antibody(“primary antibody”), the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises PD-L1, the second ligand comprisesanti-PD-L1 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, thesecond detectable reagent comprises amino ethyl carbazole (AEC), and theagent which dissolves the second detectable reagent is alcohol oracetone.

Optionally, the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises p40, the second ligand comprisesanti-p40 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-p40 antibodies, thesecond detectable reagent comprises amino ethyl carbazole (AEC), and theagent which dissolves the second detectable reagent is alcohol oracetone.

Optionally, a counterstaining agent is hematoxylin.

Optionally, the method further comprises using a digital image of thesignals detected in the biological sample to train a neural network.

Optionally, a neural network trained using the method.

Optionally, an annotated image obtained by the method.

According to an aspect of some embodiments of the present inventionthere is provided a method for detecting multiple target molecules in abiological sample comprising cells comprising: contacting the biologicalsample with one or more first reagents which generate a first detectablesignal in cells comprising a first target molecule, wherein said firstdetectable signal is detectable using a first detection method;contacting the biological sample with one or more second reagents whichgenerate a second detectable signal in cells comprising a second targetmolecule, wherein the second detectable signal is detectable using asecond detection method and is substantially undetectable using thefirst detection method and wherein the first detectable signal issubstantially undetectable using the second detection method; detectingthe signal generated by the one or more first reagents using the firstdetection method; and detecting the signal generated by the one or moresecond reagents using the second detection method.

Optionally, the biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

Optionally, the target molecules are indicative of at least one ofnormal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

Optionally, the target molecules are selected from the group consistingof nuclear proteins, cytoplasmic proteins, membrane proteins, nuclearantigens, cytoplasmic antigens, membrane antigens and nucleic acids.

Optionally, the target molecules are nucleic acids.

Optionally, the target molecules are polypeptides.

Optionally, the first reagents comprise a first primary antibody againsta first target polypeptide, an HRP coupled polymer that binds to thefirst primary antibody and a chromogen which is an HRP substrate.

Optionally, the second reagents comprise a second primary antibodyagainst a second target polypeptide, an HRP coupled polymer that bindsto the second primary antibody, and a rhodamine based fluorescentcompound coupled to a long single chained polymer.

Optionally, the method further comprises using a digital image of thesignals detected in the biological sample to train a neural network.

Optionally, there is provided a neural network trained using the method.

According to an aspect of some embodiments of the present inventionthere is provided a method for generating cell-level annotations from abiological sample comprising cells, the method comprising:

-   a) exposing the biological sample to a first ligand that recognizes    a first antigen thereby forming a first ligand antigen complex;-   b) exposing the first ligand antigen complex to a first labeling    reagent binding to the first ligand, the first labeling reagent    forming a first detectable reagent, wherein the first detectable    reagent is visible in brightfield; whereby the first detectable    reagent is precipitated around the first antigen;-   c) exposing the biological sample to a second ligand that recognizes    a second antigen thereby forming a second ligand antigen complex;-   d) exposing the second ligand antigen complex to a second labeling    reagent binding to the second ligand, the second labeling reagent    forming a second detectable reagent, wherein the second detectable    reagent is visible in fluorescence; whereby the second detectable    reagent is precipitated around the second antigen;-   e) obtaining a first brightfield image of the biological sample with    the first detectable reagent precipitated in the biological sample;-   f) obtaining a second fluorescent image of the biological sample    with the second detectable reagent precipitated in the biological    sample;-   g) creating a mask from the second image; and-   h) applying the mask to the first image so as to obtain an annotated    image of the tissue sample with the second marker.

Optionally, the method further comprises, prior to step a), applying atarget retrieval buffer and protein blocking solution to the biologicalsample, whereby the first and second antigens are exposed for thesubsequent steps and endogenous peroxidases are inactivated.

Optionally, the first biological sample comprises one of a human tissuesample, an animal tissue sample, or a plant tissue sample.

Optionally, the target molecules are indicative of at least one ofnormal cells, cell type, abnormal cells, damaged cells, cancer cells,tumors, subcellular structures, a marker indicative of or associatedwith a disease, disorder or health condition, or organ structures.

Optionally, the first and second labeling reagents comprise an enzymethat acts on a detectable reagent substrate to form the first and seconddetectable reagents, respectively.

Optionally, the first and second antigens are non-nuclear proteins.

Optionally, the method further comprises, following step b), denaturingthe first ligands to retrieve the first antigens available.

Optionally, the method further comprises, following step d),

-   i) counterstaining cell nuclei of the biological sample, and-   ii) dehydrating and mounting the sample on a slide.

Optionally, the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises PD-L1, the second ligand comprisesanti-PD-L1 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-PD-L1 antibodies, and thesecond detectable reagent comprises a rhodamine-based fluorescentcompound coupled to a long single-chain polymer.

Optionally, the first antigen comprises PD-L1, the first ligandcomprises anti-PD-L1 antibodies, the first labeling reagent comprises ahorseradish peroxidase (HRP)-coupled polymer capable of binding toanti-PD-L1 antibodies, the first detectable reagent comprises arhodamine-based fluorescent compound coupled to a long single-chainpolymer, the second labeling reagent comprises an HRP-coupled polymercapable of binding to the primary antibody, and the second detectablereagent comprises an HRP substrate comprising HRP Magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB).

Optionally, the first antigen comprises a lymphocyte-specific antigen,the first ligand comprises an anti-lymphocyte-specific antigen antibody(“primary antibody”), the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to the primaryantibody, the first detectable reagent comprises an HRP substratecomprising HRP Magenta or 3,3′-diaminobenzidine tetrahydrochloride(DAB), the second antigen comprises p40, the second ligand comprisesanti-p40 antibodies, the second labeling reagent comprises anHRP-coupled polymer capable of binding to anti-p40 antibodies, and thesecond detectable reagent comprises rhodamine-based fluorescent compoundcoupled to a long single-chain polymer.

Optionally, the first antigen comprises p40, the first ligand comprisesanti-p40 antibodies, the first labeling reagent comprises a horseradishperoxidase (HRP)-coupled polymer capable of binding to anti-p40antibodies, the first detectable reagent comprises rhodamine-basedfluorescent compound coupled to a long single-chain polymer, the secondantigen comprises a lymphocyte-specific antigen, the second ligandcomprises an anti-lymphocyte-specific antigen antibody (“primaryantibody”), the second labeling reagent comprises HRP-coupled polymercapable of binding to the primary antibody, and the second detectablereagent comprises an HRP substrate comprising HRP Magenta or3,3′-diaminobenzidine tetrahydrochloride (DAB).

Optionally, a counterstaining agent is hematoxylin.

Optionally, an annotated image obtained by the method.

According to an aspect of some embodiments of the present inventionthere is provided a method, comprising: receiving, with a computingsystem, a first image of a first biological sample, the first imagecomprising a first field of view (“FOV”) of the first biological samplethat has been stained with a first stain; receiving, with the computingsystem, a second image of the first biological sample, the second imagecomprising a second FOV of the first biological sample that has beenstained with a second stain; autonomously creating, with the computingsystem, a first set of image patches based on the first image and thesecond image, by extracting a portion of the first image and extractinga corresponding portion of the second image, the first set of imagepatches comprising a first patch corresponding to the extracted portionof the first image and a second patch corresponding to the extractedportion of the second image, wherein the first set of image patchescomprises labeling of instances of features of interest in the firstbiological sample that is based at least in part on at least one ofinformation contained in the first patch, information contained in thesecond patch, or information contained in one or more external labelingsources; training, using an artificial intelligence (“AI”) system, afirst model (“Model G*”) to generate first instance classification offeatures of interest (“Ground Truth”) in the first biological sample,based at least in part on the first set of image patches and thelabeling of instances of features of interest contained in the first setof image patches; and training, using the AI system, a second AI model(“Model G”) to identify instances of features of interest in the firstbiological sample, based at least in part on the first patch and thefirst instance classification of features of interest generated by ModelG*.

Optionally, the computing system comprises one of a computing systemdisposed in a work environment, a remote computing system disposedexternal to the work environment and accessible over a network, a webserver, a web browser, or a cloud computing system, wherein the workenvironment comprises at least one of a laboratory, a clinic, a medicalfacility, a research facility, a healthcare facility, or a room.

Optionally, the AI system comprises at least one of a machine learningsystem, a deep learning system, a model architecture, a statisticalmodel-based system, or a deterministic analysis system, wherein themodel architecture comprises at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”).

Optionally, the first biological sample comprises one of a human tissuesample, an animal tissue sample, a plant tissue sample, or anartificially produced tissue sample, wherein the features of interestcomprise at least one of normal cells, abnormal cells, damaged cells,cancer cells, tumors, subcellular structures, or organ structures.

Optionally, the first image comprises highlighting of first features ofinterest in the first biological sample by the first stain that had beenapplied to the first biological sample, and wherein the second imagecomprises one of highlighting of second features of interest in thefirst biological sample by the second stain that had been applied to thefirst biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain, the second features ofinterest being different from the first features of interest, whereinthe second stain is one of the same as the first stain but used to stainthe second features of interest or different from the first stain.

Optionally, the first stain comprises at least one of Hematoxylin,Acridine orange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet,Crystal violet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidiumbromide intercalates, Acid fuchsine, Hoechst stain, Iodine, Malachitegreen, Methyl green, Methylene blue, Neutral red, Nile blue, Nile red,Osmium tetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and wherein thesecond stain comprises at least one of Hematoxylin, Acridine orange,Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystal violet,DAPI, Eosin, Ethidium bromide intercalates, Acid fuchsine, Hoechststain, Iodine, Malachite green, Methyl green, Methylene blue, Neutralred, Nile blue, Nile red, Osmium tetroxide, Propidium Iodide, Rhodamine,Safranine, PD-L1 stain, DAB chromogen, Magenta chromogen, cyaninechromogen, CD3 stain, CD20 stain, CD68 stain, p40 stain, antibody-basedstain, or label-free imaging marker.

Optionally, the first image comprises one of a first set of color orbrightfield images, a first fluorescence image, a first phase image, ora first spectral image, wherein the first set of color or brightfieldimages comprises a first red-filtered (“R”) image, a firstgreen-filtered (“G”) image, and a first blue-filtered (“B”) image,wherein the first fluorescence image comprises at least one of a firstautofluorescence image or a first labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the first spectral image comprises at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, wherein the second image comprises one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, and wherein the second set ofcolor or brightfield images comprises a second R image, a second Gimage, and a second B image, wherein the second fluorescence imagecomprises at least one of second autofluorescence image or a secondlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, wherein the second spectralimage comprises at least one of a second Raman spectroscopy image, asecond NIR spectroscopy image, a second multispectral image, a secondhyperspectral image, or a second full spectral image.

Optionally, the method further comprises: aligning, with the computingsystem, the first image with the second image to create first set ofaligned images, by aligning one or more features of interest in thefirst biological sample as depicted in the first image with the same oneor more features of interest in the first biological sample as depictedin the second image, wherein aligning the first image with the secondimage is one of performed manually using manual inputs to the computingsystem or performed autonomously, wherein autonomous alignment mightcomprise alignment using at least one of automated global alignmenttechniques or optical flow alignment techniques.

Optionally, the method further comprises: receiving, with the computingsystem, a third image of the first biological sample, the third imagecomprising a third FOV of the first biological sample that has beenstained with a third stain; autonomously performing, with the computingsystem, one of: aligning the third image with each of the first imageand the second image to create second aligned images, by aligning one ormore features of interest in the first biological sample as depicted inthe third image with the same one or more features of interest in thefirst biological sample as depicted in each of the first image and thesecond image; or aligning the third image with the first aligned imagesto create second aligned images, by aligning one or more features ofinterest in the first biological sample as depicted in the third imagewith the same one or more features of interest in the first biologicalsample as depicted in the first aligned images; autonomously creating,with the computing system, second aligned image patches from the secondaligned images, by extracting a portion of the second aligned images,the portion of the second aligned images comprising a first patchcorresponding to the extracted portion of the first image, a secondpatch corresponding to the extracted portion of the second image, and athird patch corresponding to the extracted portion of the third image;training the AI system to update Model G* to generate second instanceclassification of features of interest in the first biological sample,based at least in part on the second aligned image patches and thelabeling of instances of features of interest contained in the first setof image patches; and training the AI system to update Model G toidentify instances of features of interest in the first biologicalsample, based at least in part on the first patch and the secondinstance classification of features of interest generated by Model G*.

Optionally, the first image comprises highlighting of first features ofinterest in the first biological sample by the first stain that had beenapplied to the first biological sample, wherein the second imagecomprises one of highlighting of second features of interest in thefirst biological sample by the second stain that had been applied to thefirst biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain, the second features ofinterest being different from the first features of interest, andwherein the third image comprises one of highlighting of third featuresof interest in the first biological sample by the third stain that hadbeen applied to the first biological sample in addition to highlightingof the first features of interest by the first stain and highlighting ofthe second features of interest by the second stain, highlighting of thethird features of interest by the third stain in addition to only one ofhighlighting of the first features of interest by the first stain orhighlighting of the second features of interest by the second stain, orhighlighting of the third features of interest by the third stainwithout any of highlighting of the first features of interest by thefirst stain or highlighting of the second features of interest by thesecond stain, the third features of interest being different from eachof the first features of interest and the second features of interest.

Optionally, the first image comprises one of a first set of color orbrightfield images, a first fluorescence image, a first phase image, ora first spectral image, wherein the first set of color or brightfieldimages comprises a first red-filtered (“R”) image, a firstgreen-filtered (“G”) image, and a first blue-filtered (“B”) image,wherein the first fluorescence image comprises at least one of firstautofluorescence image or a first labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the first spectral image comprises at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, wherein the second image comprises one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, wherein the second set of coloror brightfield images comprises a second R image, a second G image, anda second B image, wherein the second fluorescence image comprises atleast one of second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, wherein the second spectral image comprisesat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, wherein the third imagecomprises one of a third set of color or brightfield images, a thirdfluorescence image, a third phase image, or a third spectral image, andwherein the third set of color or brightfield images comprises a third Rimage, a third G image, and a third B image, wherein the thirdfluorescence image comprises at least one of a third autofluorescenceimage or a third labelled fluorescence image having one or more channelswith different excitation or emission characteristics, wherein the thirdspectral image comprises at least one of a third Raman spectroscopyimage, a third NIR spectroscopy image, a third multispectral image, athird hyperspectral image, or a third full spectral image.

Optionally, the method further comprises: receiving, with the computingsystem, a fourth image, the fourth image comprising one of a fourth FOVof the first biological sample different from the first FOV and thesecond FOV or a fifth FOV of a second biological sample, wherein thesecond biological sample is different from the first biological sample;and identifying, using Model G, second instances of features of interestin the second biological sample, based at least in part on the fourthimage and based at least in part on training of Model G using the firstpatch and the first instance classification of features of interestgenerated by Model G*.

Optionally, the fourth image comprises the fifth FOV and furthercomprises only highlighting of first features of interest in the secondbiological sample by the first stain that had been applied to the secondbiological sample.

Optionally, the method further comprises: generating, using Model G, aclinical score, based at least in part on the identified secondinstances of features of interest.

According to an aspect of some embodiments of the present inventionthere is provided a system, comprising:

-   a computing system, comprising:-   at least one first processor; and a first non-transitory computer    readable medium communicatively coupled to the at least one first    processor, the first non-transitory computer readable medium having    stored thereon computer software comprising a first set of    instructions that, when executed by the at least one first    processor, causes the computing system to: receive a first image of    a first biological sample, the first image comprising a first field    of view (“FOV”) of the first biological sample that has been stained    with a first stain; receive a second image of the first biological    sample, the second image comprising a second FOV of the first    biological sample that has been stained with a second stain;    autonomously create a first set of image patches based on the first    image and the second image, by extracting a portion of the first    image and extracting a corresponding portion of the second image,    the first set of image patches comprising a first patch    corresponding to the extracted portion of the first image and a    second patch corresponding to the extracted portion of the second    image, wherein the first set of image patches comprises labeling of    instances of features of interest in the first biological sample    that is based at least in part on at least one of information    contained in the first patch, information contained in the second    patch, or information contained in one or more external labeling    sources; train, using an artificial intelligence (“AI”) system, a    first model (“Model G*”) to generate first instance classification    of features of interest (“Ground Truth”) in the first biological    sample, based at least in part on the first set of image patches and    the labeling of instances of features of interest contained in the    first set of image patches; and train, using the AI system, a second    AI model (“Model G”) to identify instances of features of interest    in the first biological sample, based at least in part on the first    patch and the first instance classification of features of interest    generated by Model G*.

Optionally, the computing system comprises one of a computing systemdisposed in a work environment, a remote computing system disposedexternal to the work environment and accessible over a network, a webserver, a web browser, or a cloud computing system, wherein the workenvironment comprises at least one of a laboratory, a clinic, a medicalfacility, a research facility, a healthcare facility, or a room.

Optionally, the AI system comprises at least one of a machine learningsystem, a deep learning system, a model architecture, a statisticalmodel-based system, or a deterministic analysis system, wherein themodel architecture comprises at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”).

Optionally, the first biological sample comprises one of a human tissuesample, an animal tissue sample, a plant tissue sample, or anartificially produced tissue sample, wherein the features of interestcomprise at least one of normal cells, abnormal cells, damaged cells,cancer cells, tumors, subcellular structures, or organ structures.

Optionally, the first image comprises highlighting of first features ofinterest in the first biological sample by the first stain that had beenapplied to the first biological sample, and wherein the second imagecomprises one of highlighting of second features of interest in thefirst biological sample by the second stain that had been applied to thefirst biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain, the second features ofinterest being different from the first features of interest, whereinthe second stain is one of the same as the first stain but used to stainthe second features of interest or different from the first stain.

Optionally, the first image comprises one of a first set of color orbrightfield images, a first fluorescence image, a first phase image, ora first spectral image, wherein the first set of color or brightfieldimages comprises a first red-filtered (“R”) image, a firstgreen-filtered (“G”) image, and a first blue-filtered (“B”) image,wherein the first fluorescence image comprises at least one of a firstautofluorescence image or a first labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the first spectral image comprises at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, wherein the second image comprises one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, and wherein the second set ofcolor or brightfield images comprises a second R image, a second Gimage, and a second B image, wherein the second fluorescence imagecomprises at least one of a second autofluorescence image or a secondlabelled fluorescence image having one or more channels with differentexcitation or emission characteristics, wherein the second spectralimage comprises at least one of a second Raman spectroscopy image, asecond NIR spectroscopy image, a second multispectral image, a secondhyperspectral image, or a second full spectral image.

Optionally, the first set of instructions, when executed by the at leastone first processor, further causes the computing system to: receive athird image of the first biological sample, the third image comprising athird FOV of the first biological sample that has been stained with athird stain different from each of the first stain and the second stain;autonomously perform one of: aligning the third image with each of thefirst image and the second image images to create second aligned images,by aligning one or more features of interest in the first biologicalsample as depicted in the third image with the same one or more featuresof interest in the first biological sample as depicted in each of thefirst image and the second image; or aligning the third image with thefirst aligned images to create second aligned images, by aligning one ormore features of interest in the first biological sample as depicted inthe third image with the same one or more features of interest in thefirst biological sample as depicted in the first aligned images;autonomously create second aligned image patches from the second alignedimages, by extracting a portion of the second aligned images, theportion of the second aligned images comprising a first patchcorresponding to the extracted portion of the first image, a secondpatch corresponding to the extracted portion of the second image, and athird patch corresponding to the extracted portion of the third image;train the AI system to update Model G* to generate second instanceclassification of features of interest in the first biological sample,based at least in part on the second aligned image patches and thelabeling of instances of features of interest contained in the first setof image patches; and train the AI system to update Model G to identifyinstances of features of interest in the first biological sample, basedat least in part on the first patch and the second instanceclassification of features of interest generated by Model G*.

Optionally, the first set of instructions, when executed by the at leastone first processor, further causes the computing system to: receive afourth image, the fourth image comprising one of a fourth FOV of thefirst biological sample different from the first FOV and the second FOVor a fifth FOV of a second biological sample, wherein the secondbiological sample is different from the first biological sample; andidentify, using Model G, second instances of features of interest in thesecond biological sample, based at least in part on the fourth image andbased at least in part on training of Model G using the first patch andthe first instance classification of features of interest generated byModel G*.

According to an aspect of some embodiments of the present inventionthere is provided a method, comprising: receiving, with a computingsystem, a first image of a first biological sample, the first imagecomprising a first field of view (“FOV”) of the first biological sample;and identifying, using a first artificial intelligence (“AI”) model(“Model G”) that is generated or updated by a trained AI system, firstinstances of features of interest in the first biological sample, basedat least in part on the first image and based at least in part ontraining of Model G using a first patch and first instanceclassification of features of interest generated by a second model(Model G*) that is generated or updated by the trained system by usingfirst aligned image patches and labeling of instances of features ofinterest contained in the first patch, wherein the first aligned imagepatches comprise an extracted portion of first aligned images, whereinthe first aligned images comprise a second image and a third image thathave been aligned, wherein the second image comprises a second FOV of asecond biological sample that has been stained with a first stain, thesecond biological sample being different from the first biologicalsample, wherein the third image comprises a third FOV of the secondbiological sample that has been stained with a second stain, wherein thesecond patch comprises labeling of instances of features of interest asshown in the extracted portion of the second image of the first alignedimages, wherein the first image comprises highlighting of first featuresof interest in the second biological sample by the first stain that hadbeen applied to the second biological sample, wherein the second patchcomprises one of highlighting of second features of interest in thesecond biological sample by the second stain that had been applied tothe second biological sample in addition to highlighting of the firstfeatures of interest by the first stain or highlighting of the secondfeatures of interest by the second stain without highlighting of thefirst features of interest by the first stain.

According to an aspect of some embodiments of the present inventionthere is provided a system, comprising:

-   a computing system, comprising:-   at least one first processor; and a first non-transitory computer    readable medium communicatively coupled to the at least one first    processor, the first non-transitory computer readable medium having    stored thereon computer software comprising a first set of    instructions that, when executed by the at least one first    processor, causes the computing system to: receive a first image of    a first biological sample, the first image comprising a first field    of view (“FOV”) of the first biological sample; and identify, using    a first artificial intelligence (“AI”) model (“Model G”) that is    generated or updated by a trained AI system, first instances of    features of interest in the first biological sample, based at least    in part on the first image and based at least in part on training of    Model G using a first patch and first instance classification of    features of interest generated by a second model (Model G*) that is    generated or updated by the trained AI system by using first aligned    image patches and labeling of instances of features of interest    contained in the first patch, wherein the first aligned image    patches comprise an extracted portion of first aligned images,    wherein the first aligned images comprise a second image and a third    image that have been aligned, wherein the second image comprises a    second FOV of a second biological sample that has been stained with    a first stain, the second biological sample being different from the    first biological sample, wherein the third image comprises a third    FOV of the second biological sample that has been stained with a    second stain, wherein the second patch comprises labeling of    instances of features of interest as shown in the extracted portion    of the second image of the first aligned images, wherein the first    image comprises highlighting of first features of interest in the    second biological sample by the first stain that had been applied to    the second biological sample wherein the second patch comprises one    of highlighting of second features of interest in the second    biological sample by the second stain that had been applied to the    second biological sample in addition to highlighting of the first    features of interest by the first stain or highlighting of the    second features of interest by the second stain without highlighting    of the first features of interest by the first stain.

According to an aspect of some embodiments of the present inventionthere is provided a method, comprising: receiving, with a computingsystem, first instance classification of features of interest (“GroundTruth”) in a first biological sample that has been sequentially stained,the Ground Truth having been generated by a trained first model (“ModelG*”) that has been trained or updated by an artificial intelligence(“AI”) system, wherein the Ground Truth is generated by using firstaligned image patches and labeling of instances of features of interestcontained in the first aligned image patches, wherein the first alignedimage patches comprise an extracted portion of first aligned images,wherein the first aligned images comprise a first image and a secondimage that have been aligned, wherein the first image comprises a firstFOV of the first biological sample that has been stained with a firststain, wherein the second image comprises a second FOV of the firstbiological sample that has been stained with a second stain, wherein thefirst aligned image patches comprise labeling of instances of featuresof interest as shown in the extracted portion of the first alignedimages; and training, using the AI system, a second AI model (“Model G”)to identify instances of features of interest in the first biologicalsample, based at least in part on the first instance classification offeatures of interest generated by Model G*.

According to an aspect of some embodiments of the present inventionthere is provided a system, comprising:

-   a computing system, comprising:-   at least one first processor; and a first non-transitory computer    readable medium communicatively coupled to the at least one first    processor, the first non-transitory computer readable medium having    stored thereon computer software comprising a first set of    instructions that, when executed by the at least one first    processor, causes the computing system to: receive first instance    classification of features of interest (“Ground Truth”) in a first    biological sample that has been sequentially stained, the Ground    Truth having been generated by a trained first model (“Model G*”)    that has been trained or updated by an artificial intelligence    (“AI”) system, wherein the Ground Truth is generated by using first    aligned image patches and labeling of instances of features of    interest contained in the first aligned image patches, wherein the    first aligned image patches comprise an extracted portion of first    aligned images, wherein the first aligned images comprise a first    image and a second image that have been aligned, wherein the first    image comprises a first FOV of the first biological sample that has    been stained with a first stain, wherein the second image comprises    a second FOV of the first biological sample that has been stained    with a second stain, wherein the first aligned image patches    comprise labeling of instances of features of interest as shown in    the extracted portion of the first aligned images; and train, using    the AI system, a second AI model (“Model G”) to identify instances    of features of interest in the first biological sample, based at    least in part on the first. instance classification of features of    interest generated by Model G*.

According to an aspect of some embodiments of the present inventionthere is provided a method, comprising: generating, using a computingsystem, ground truth for developing accurate artificial intelligence(“AI”) models for biological image interpretation, based at least inpart on images of a first biological sample depicting sequentialstaining of the first biological sample.

According to an aspect of some embodiments of the present inventionthere is provided a method, comprising: receiving, with a computingsystem, a first image of a first biological sample, the first imagecomprising a first field of view (“FOV”) of the first biological samplethat has been stained with a first stain; receiving, with the computingsystem, a second image of the first biological sample, the second imagecomprising a second FOV of the first biological sample that has beenstained with at least a second stain; aligning, with the computingsystem, the first image with the second image to create first alignedimages, by aligning one or more features of interest in the firstbiological sample as depicted in the first image with the same one ormore features of interest in the first biological sample as depicted inthe second image; autonomously creating, with the computing system,first aligned image patches from the first aligned images, by extractinga portion of the first aligned images, the portion of the first alignedimages comprising a first patch corresponding to the extracted portionof the first image and a second patch corresponding to the extractedportion of the second image; training, using an artificial intelligence(“AI”) system, a first AI model (“Model F”) to generate a third patchcomprising a virtual stain of the first aligned image patches, based atleast in part on the first patch and the second patch, the virtual stainsimulating staining by at least the second stain of features of interestin the first biological sample as shown in the second patch; andtraining, using the AI system, a second model (“Model G*”) to identifyor classify first instances of features of interest in the firstbiological sample, based at least in part on the third patch and basedat least in part on results from an external instance classificationprocess or a region of interest detection process.

Optionally, the computing system comprises one of a computing systemdisposed in a work environment, a remote computing system disposedexternal to the work environment and accessible over a network, a webserver, a web browser, or a cloud computing system, wherein the workenvironment comprises at least one of a laboratory, a clinic, a medicalfacility, a research facility, a healthcare facility, or a room.

Optionally, the AI system comprises at least one of a machine learningsystem, a deep learning system, a model architecture, a statisticalmodel-based system, or a deterministic analysis system, wherein themodel architecture comprises at least one of a neural network, aconvolutional neural network (“CNN”), or a fully convolutional network(“FCN”).

Optionally, the first biological sample comprises one of a human tissuesample, an animal tissue sample, a plant tissue sample, or anartificially produced tissue sample, wherein the features of interestcomprise at least one of normal cells, abnormal cells, damaged cells,cancer cells, tumors, subcellular structures, or organ structures.

Optionally, the first image comprises highlighting of first features ofinterest in the first biological sample by the first stain that had beenapplied to the first biological sample, and wherein the third patchcomprises one of highlighting of the first features of interest by thefirst stain and highlighting of second features of interest in the firstbiological sample by the virtual stain that simulates the second stainhaving been applied to the first biological sample or highlighting ofthe first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest, whereinthe second stain is one of the same as the first stain but used to stainthe second features of interest or different from the first stain.

Optionally, the first stain comprises at least one of Hematoxylin,Acridine orange, Bismarck brown, Carmine, Coomassie blue, Cresyl violet,Crystal violet, 4′,6-diamidino-2-phenylindole (“DAPI”), Eosin, Ethidiumbromide intercalates, Acid fuchsine, Hoechst stain, Iodine, Malachitegreen, Methyl green, Methylene blue, Neutral red, Nile blue, Nile red,Osmium tetroxide, Propidium Iodide, Rhodamine, Safranine, programmeddeath-ligand 1 (“PD-L1”) stain, 3,3′-Diaminobenzidine (“DAB”) chromogen,Magenta chromogen, cyanine chromogen, cluster of differentiation (“CD”)3 stain, CD20 stain, CD68 stain, 40S ribosomal protein SA (“p40”) stain,antibody-based stain, or label-free imaging marker, and wherein thesecond stain comprises at least one of Hematoxylin, Acridine orange,Bismarck brown, Carmine, Coomassie blue, Cresyl violet, Crystal violet,DAPI, Eosin, Ethidium bromide intercalates, Acid fuchsine, Hoechststain, Iodine, Malachite green, Methyl green, Methylene blue, Neutralred, Nile blue, Nile red, Osmium tetroxide, Propidium Iodide, Rhodamine,Safranine, PD-L1 stain, DAB chromogen, Magenta chromogen, cyaninechromogen, CD3 stain, CD20 stain, CD68 stain, p40 stain, antibody-basedstain, or label-free imaging marker.

Optionally, the first image comprises one of a first set of color orbrightfield images, a first fluorescence image, a first phase image, ora first spectral image, wherein the first set of color or brightfieldimages comprises a first red-filtered (“R”) image, a firstgreen-filtered (“G”) image, and a first blue-filtered (“B”) image,wherein the first fluorescence image comprises at least one of a firstautofluorescence image or a first labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the first spectral image comprises at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, wherein the second image comprises one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, wherein the second set of coloror brightfield images comprises a second R image, a second G image, anda second B image, wherein the second fluorescence image comprises atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, wherein the second spectral image comprisesat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, wherein the third patchcomprises one of a third set of color or brightfield images, a thirdfluorescence image, a third phase image, or a third spectral image, andwherein the third set of color or brightfield images comprises a third Rimage, a third G image, and a third B image, wherein the thirdfluorescence image comprises at least one of a third autofluorescenceimage or a third labelled fluorescence image having one or more channelswith different excitation or emission characteristics, wherein the thirdspectral image comprises at least one of a third Raman spectroscopyimage, a third NIR spectroscopy image, a third multispectral image, athird hyperspectral image, or a third full spectral image.

Optionally, aligning the first image with the second image is one ofperformed manually using manual inputs to the computing system orperformed autonomously, wherein autonomous alignment might comprisealignment using at least one of automated global alignment techniques oroptical flow alignment techniques.

Optionally, training, using the AI system, Model G* to identify orclassify instances of features of interest in the first biologicalsample comprises training, using the AI system, Model G* to identify orclassify instances of features of interest in the first biologicalsample, based at least in part on one or more of the first patch, thesecond patch, or the third patch and based at least in part on theresults from the external instance classification process or the regionof interest detection process.

Optionally, the external instance classification process or the regionof interest detection process each comprises at least one of detectionof nuclei in the first image or the first patch by a nuclei detectionmethod, identification of nuclei in the first image or the first patchby a pathologist, detection of features of interest in the first imageor the first patch by a feature detection method, or identification offeatures of interest in the first image or the first patch by thepathologist.

Optionally, training the AI system to update Model F comprises:receiving, with an encoder, the first patch; receiving the second patch;encoding, with the encoder, the received first patch; decoding, with thedecoder, the encoded first patch; generating an intensity map based onthe decoded first patch; simultaneously operating on the encoded firstpatch to generate a color vector; combining the generated intensity mapwith the generated color vector to generate an image of the virtualstain; generating a predicted virtually stained image patch; determininga first loss value between the predicted virtually stained image patchand the second patch; calculating a loss value using a loss function,based on the first loss value between the predicted virtually stainedimage patch and the second patch; and updating, with the AI system,Model F to generate the third patch, by updating one or more parametersof Model F based on the calculated loss value.

Optionally, the loss function comprises one of a mean squared error lossfunction, a mean squared logarithmic error loss function, a meanabsolute error loss function, a Huber loss function, or a weighted sumof squared differences loss function.

Optionally, the first patch comprises one of a first set of color orbrightfield images, a first fluorescence image, a first phase image, ora first spectral image, wherein the first set of color or brightfieldimages comprises a first red-filtered (“R”) image, a firstgreen-filtered (“G”) image, and a first blue-filtered (“B”) image,wherein the first fluorescence image comprises at least one of a firstautofluorescence image or a first labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the first spectral image comprises at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, wherein the second image comprises one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, wherein the second set of coloror brightfield images comprises a second R image, a second G image, anda second B image, wherein the second fluorescence image comprises atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, wherein the second spectral image comprisesat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, wherein the predicted virtuallystained image patch comprises one of a third set of color or brightfieldimages, a third fluorescence image, a third phase image, or a thirdspectral image, wherein the third set of color or brightfield imagescomprises a third R image, a third G image, and a third B image, whereinthe third fluorescence image comprises at least one of a thirdautofluorescence image or a third labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the third spectral image comprises at least one of a third Ramanspectroscopy image, a third NIR spectroscopy image, a thirdmultispectral image, a third hyperspectral image, or a third fullspectral image.

Optionally, operating on the encoded first patch to generate the colorvector comprises operating on the encoded first patch to generate acolor vector, using a color vector output and one of a neural network, amodel architecture, a statistical model-based system, or a deterministicanalysis system, wherein the model architecture comprises at least oneof a neural network, a convolutional neural network (“CNN”), or a fullyconvolutional network (“FCN”), wherein the color vector is one of afixed color vector or a learned color vector that is not based on theencoded first patch.

Optionally, the image of the virtual stain comprises one of a 3-channelimage of the virtual stain, a RGB-transform image of the virtual stain,or a logarithmic transform image of the virtual stain.

Optionally, generating the predicted virtually stained image patchcomprises adding the generated image of the virtual stain to thereceived first patch to produce the predicted virtually stained imagepatch.

Optionally, the first loss value comprises one of a pixel loss valuebetween each pixel in the predicted virtually stained image patch and acorresponding pixel in the second patch or a generative adversarialnetwork (“GAN”) loss value between the predicted virtually stained imagepatch and the second patch, wherein the GAN loss value is generatedbased on one of a minimax GAN loss function, a non-saturating GAN lossfunction, a least squares GAN loss function, or a Wasserstein GAN lossfunction.

Optionally, training the AI system to update Model F comprises:receiving, with the AI system, the first patch; receiving, with the AIsystem, the second patch; generating, with a second model of the AIsystem, an image of the virtual stain; adding the generated image of thevirtual stain to the received first patch to produce a predictedvirtually stained image patch; determining a first loss value betweenthe predicted virtually stained image patch and the second patch;calculating a loss value using a loss function, based on the first lossvalue between the predicted virtually stained image patch and the secondpatch; and updating, with the AI system, Model F to generate the thirdpatch, by updating one or more parameters of Model F based on thecalculated loss value.

Optionally, the second model comprises at least one of a convolutionalneural network (“CNN”), a U-Net, an artificial neural network (“ANN”), aresidual neural network (“ResNet”), an encode/decode CNN, anencode/decode U-Net, an encode/decode ANN, or an encode/decode ResNet.

Optionally, the method further comprises: receiving, with the computingsystem, a fourth image, the fourth image comprising one of a fourth FOVof the first biological sample different from the first FOV and thesecond FOV or a fifth FOV of a second biological sample, wherein thesecond biological sample is different from the first biological sample;and; and identifying, using Model G*, second instances of features ofinterest in the second biological sample, based at least in part on thefourth image and based at least in part on training of Model G* using atleast the third patch comprising the virtual stain of the first alignedimage patches. Optionally, the method further comprises: generating,using Model G*, a clinical score, based at least in part on theidentified second instances of features of interest.

According to an aspect of some embodiments of the present inventionthere is provided a system, comprising:

-   a computing system, comprising:-   at least one first processor; and a first non-transitory computer    readable medium communicatively coupled to the at least one first    processor, the first non-transitory computer readable medium having    stored thereon computer software comprising a first set of    instructions that, when executed by the at least one first    processor, causes the computing system to: receive a first image of    a first biological sample, the first image comprising a first field    of view (“FOV”) of the first biological sample that has been stained    with a first stain; receive a second image of the first biological    sample, the second image comprising a second FOV of the first    biological sample that has been stained with at least a second    stain; align the first image with the second image to create first    aligned images, by aligning one or more features of interest in the    first biological sample as depicted in the first image with the same    one or more features of interest in the first biological sample as    depicted in the second image; autonomously create first aligned    image patches from the first aligned images, by extracting a portion    of the first aligned images, the portion of the first aligned images    comprising a first patch corresponding to the extracted portion of    the first image and a second patch corresponding to the extracted    portion of the second image; train, using an artificial intelligence    (“AI”) system, a first AI model (“Model F”) to generate a third    patch comprising a virtual stain of the first aligned image patches,    based at least in part on the first patch and the second patch, the    virtual stain simulating staining by at least the second stain of    features of interest in the first biological sample as shown in the    second patch; and train, using the AI system, a second model (“Model    G*”) to identify or classify first instances of features of interest    in the first biological sample, based at least in part on the third    patch and based at least in part on results from an external    instance classification process or a region of interest detection    process.

Optionally, the computing system comprises one of a computing systemdisposed in a work environment, a remote computing system disposedexternal to the work environment and accessible over a network, a webserver, a web browser, or a cloud computing system, wherein the workenvironment comprises at least one of a laboratory, a clinic, a medicalfacility, a research facility, a healthcare facility, or a room.

Optionally, the AI system comprises at least one of a machine learningsystem, a deep learning system, a neural network, a model architecture,a statistical model-based system, or a deterministic analysis system,wherein the model architecture comprises at least one of a neuralnetwork, a convolutional neural network (“CNN”), or a fullyconvolutional network (“FCN”).

Optionally, the first biological sample comprises one of a human tissuesample, an animal tissue sample, a plant tissue sample, or anartificially produced tissue sample, wherein the features of interestcomprise at least one of normal cells, abnormal cells, damaged cells,cancer cells, tumors, subcellular structures, or organ structures.

Optionally, the first image comprises highlighting of first features ofinterest in the first biological sample by the first stain that had beenapplied to the first biological sample, and wherein the third patchcomprises one of highlighting of the first features of interest by thefirst stain and highlighting of second features of interest in the firstbiological sample by the virtual stain that simulates the second stainhaving been applied to the first biological sample or highlighting ofthe first features of interest by the first stain and highlighting offirst features of interest by the virtual stain, the second features ofinterest being different from the first features of interest, whereinthe second stain is one of the same as the first stain but used to stainthe second features of interest or different from the first stain.Optionally, the first image comprises one of a first set of color orbrightfield images, a first fluorescence image, a first phase image, ora first spectral image, wherein the first set of color or brightfieldimages comprises a first red-filtered (“R”) image, a firstgreen-filtered (“G”) image, and a first blue-filtered (“B”) image,wherein the first fluorescence image comprises at least one of a firstautofluorescence image or a first labelled fluorescence image having oneor more channels with different excitation or emission characteristics,wherein the first spectral image comprises at least one of a first Ramanspectroscopy image, a first near infrared (“NIR”) spectroscopy image, afirst multispectral image, a first hyperspectral image, or a first fullspectral image, wherein the second image comprises one of a second setof color or brightfield images, a second fluorescence image, a secondphase image, or a second spectral image, wherein the second set of coloror brightfield images comprises a second R image, a second G image, anda second B image, wherein the second fluorescence image comprises atleast one of a second autofluorescence image or a second labelledfluorescence image having one or more channels with different excitationor emission characteristics, wherein the second spectral image comprisesat least one of a second Raman spectroscopy image, a second NIRspectroscopy image, a second multispectral image, a second hyperspectralimage, or a second full spectral image, wherein the third patchcomprises one of a third set of color or brightfield images, a thirdfluorescence image, a third phase image, or a third spectral image, andwherein the third set of color or brightfield images comprises a third Rimage, a third G image, and a third B image, wherein the thirdfluorescence image comprises at least one of a third autofluorescenceimage or a third labelled fluorescence image having one or more channelswith different excitation or emission characteristics, wherein the thirdspectral image comprises at least one of a third Raman spectroscopyimage, a third NIR spectroscopy image, a third multispectral image, athird hyperspectral image, or a third full spectral image.

Optionally, training the AI system to update Model F comprises:receiving, with an encoder, the first patch; receiving the second patch;encoding, with the encoder, the received first patch; decoding, with thedecoder, the encoded first patch; generating an intensity map based onthe decoded first patch; simultaneously operating on the encoded firstpatch to generate a color vector; combining the generated intensity mapwith the generated color vector to generate an image of the virtualstain; adding the generated image of the virtual stain to the receivedfirst patch to produce a predicted virtually stained image patch;determining a first loss value between the predicted virtually stainedimage patch and the second patch; calculating a loss value using a lossfunction, based on the first loss value between the predicted virtuallystained image patch and the second patch; and updating, with the AIsystem, Model F to generate the third patch, by updating one or moreparameters of Model F based on the calculated loss value.

Optionally, the loss function comprises one of a mean squared error lossfunction, a mean squared logarithmic error loss function, a meanabsolute error loss function, a Huber loss function, or a weighted sumof squared differences loss function.

Optionally, the first loss value comprises one of a pixel loss valuebetween each pixel in the predicted virtually stained image patch and acorresponding pixel in the second patch or a generative adversarialnetwork (“GAN”) loss value between the predicted virtually stained imagepatch and the second patch, wherein the GAN loss value may be generatedbased on one of a minimax GAN loss function, a non-saturating GAN lossfunction, a least squares GAN loss function, or a Wasserstein GAN lossfunction.

Optionally, the first set of instructions, when executed by the at leastone first processor, further causes the computing system to: receive afourth image, the fourth image comprising one of a fourth FOV of thefirst biological sample different from the first FOV and the second FOVor a fifth FOV of a second biological sample, wherein the secondbiological sample is different from the first biological sample; andidentify, using Model G*, second instances of features of interest inthe second biological sample, based at least in part on the fourth imageand based at least in part on training of Model G* using at least thethird patch comprising the virtual stain of the first aligned imagepatches.

According to an aspect of some embodiments of the present inventionthere is provided a method, comprising: receiving, with a computingsystem, a first image of a first biological sample, the first imagecomprising a first field of view (“FOV”) of the first biological sample;and identifying, using a first model (“Model G*”) that is generated orupdated by a trained artificial intelligence (“AI”) system, firstinstances of features of interest in the first biological sample, basedat least in part on the first image and based at least in part ontraining of Model G* using at least a first patch comprising a virtualstain of first aligned image patches, the first patch being generated bya second AI model (Model F) that is generated or updated by the trainedAI system by using a second patch, wherein the first aligned imagepatches comprise an extracted portion of first aligned images, whereinthe first aligned images comprise a second image and a third image thathave been aligned, wherein the second image comprises a second FOV of asecond biological sample that is different from the first biologicalsample that has been stained with a first stain, wherein the secondpatch comprises the extracted portion of the second image, wherein thethird image comprises a third FOV of the second biological sample thathas been stained with at least a second stain.

According to an aspect of some embodiments of the present inventionthere is provided a system, comprising:

-   a computing system, comprising:-   at least one first processor; and a first non-transitory computer    readable medium communicatively coupled to the at least one first    processor, the first non-transitory computer readable medium having    stored thereon computer software comprising a first set of    instructions that, when executed by the at least one first    processor, causes the computing system to: receive a first image of    a first biological sample, the first image comprising a first field    of view (“FOV”) of the first biological sample; and identify, using    a first model (“Model G*”) that is generated or updated by a trained    artificial intelligence (“AI”) system, first instances of features    of interest in the first biological sample, based at least in part    on the first image and based at least in part on training of Model    G* using at least a first patch comprising a virtual stain of first    aligned image patches, the first patch being generated by a second    AI model (Model F) that is generated or updated by the trained AI    system by using a second patch, wherein the first aligned image    patches comprise an extracted portion of first aligned images,    wherein the first aligned images comprise a second image and a third    image that have been aligned, wherein the second image comprises a    second FOV of a second biological sample that is different from the    first biological sample that has been stained with a first stain,    wherein the second patch comprises the extracted portion of the    second image, wherein the third image comprises a third FOV of the    second biological sample that has been stained with at least a    second stain.

While certain features and aspects have been described with respect toexemplary embodiments, one skilled in the art will recognize thatnumerous modifications are possible. For example, the methods andprocesses described herein may be implemented using hardware components,software components, and/or any combination thereof. Further, whilevarious methods and processes described herein may be described withrespect to particular structural and/or functional components for easeof description, methods provided by various embodiments are not limitedto any particular structural and/or functional architecture but insteadcan be implemented on any suitable hardware, firmware and/or softwareconfiguration. Similarly, while certain functionality is ascribed tocertain system components, unless the context dictates otherwise, thisfunctionality can be distributed among various other system componentsin accordance with the several embodiments.

Moreover, while the procedures of the methods and processes describedherein are described in a particular order for ease of description,unless the context dictates otherwise, various procedures may bereordered, added, and/or omitted in accordance with various embodiments.Moreover, the procedures described with respect to one method or processmay be incorporated within other described methods or processes;likewise, system components described according to a particularstructural architecture and/or with respect to one system may beorganized in alternative structural architectures and/or incorporatedwithin other described systems. Hence, while various embodiments aredescribed with—or without— certain features for ease of description andto illustrate exemplary aspects of those embodiments, the variouscomponents and/or features described herein with respect to a particularembodiment can be substituted, added and/or subtracted from among otherdescribed embodiments, unless the context dictates otherwise.Consequently, although several exemplary embodiments are describedabove, it will be appreciated that the invention is intended to coverall modifications and equivalents within the scope of the followingclaims.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

It is expected that during the life of a patent maturing from thisapplication many relevant machine learning models will be developed andthe scope of the term machine learning model is intended to include allsuch new technologies a priori.

As used herein the term “about” refers to ± 10%.

The terms “comprises”, “comprising”, “includes”, “including”, “having”and their conjugates mean “including but not limited to”. This termencompasses the terms “consisting of” and “consisting essentially of”.

The phrase “consisting essentially of” means that the composition ormethod may include additional ingredients and/or steps, but only if theadditional ingredients and/or steps do not materially alter the basicand novel characteristics of the claimed composition or method.

As used herein, the singular form “a”, “an” and “the” include pluralreferences unless the context clearly dictates otherwise. For example,the term “a compound” or “at least one compound” may include a pluralityof compounds, including mixtures thereof.

The word “exemplary” is used herein to mean “serving as an example,instance or illustration”. Any embodiment described as “exemplary” isnot necessarily to be construed as preferred or advantageous over otherembodiments and/or to exclude the incorporation of features from otherembodiments.

The word “optionally” is used herein to mean “is provided in someembodiments and not provided in other embodiments”. Any particularembodiment of the invention may include a plurality of “optional”features unless such features conflict.

Throughout this application, various embodiments of this invention maybe presented in a range format. It should be understood that thedescription in range format is merely for convenience and brevity andshould not be construed as an inflexible limitation on the scope of theinvention. Accordingly, the description of a range should be consideredto have specifically disclosed all the possible subranges as well asindividual numerical values within that range. For example, descriptionof a range such as from 1 to 6 should be considered to have specificallydisclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numberswithin that range, for example, 1, 2, 3, 4, 5, and 6. This appliesregardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to includeany cited numeral (fractional or integral) within the indicated range.The phrases “ranging/ranges between” a first indicate number and asecond indicate number and “ranging/ranges from” a first indicate number“to” a second indicate number are used herein interchangeably and aremeant to include the first and second indicated numbers and all thefractional and integral numerals therebetween.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination or as suitable in any other describedembodiment of the invention. Certain features described in the contextof various embodiments are not to be considered essential features ofthose embodiments, unless the embodiment is inoperative without thoseelements.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims.

It is the intent of the applicant(s) that all publications, patents andpatent applications referred to in this specification are to beincorporated in their entirety by reference into the specification, asif each individual publication, patent or patent application wasspecifically and individually noted when referenced that it is to beincorporated herein by reference. In addition, citation oridentification of any reference in this application shall not beconstrued as an admission that such reference is available as prior artto the present invention. To the extent that section headings are used,they should not be construed as necessarily limiting. In addition, anypriority document(s) of this application is/are hereby incorporatedherein by reference in its/their entirety.

What is claimed is:
 1. A computer implemented method for training avirtual stainer machine learning model, comprising: creating an imagingmulti-record training dataset, wherein a record comprises: a first imageof a sample of tissue of a subject stained with a removable stain, and aground truth indicated by a second image of the sample of tissue of thesubject stained with a permanent stain; and training a virtual stainermachine learning model on the imaging multi-record training dataset forgenerating a virtual image depicting the permanent stain in response toan input image depicting the removable stain.
 2. The computerimplemented method of claim 1, wherein the removable stain is selectedfrom a group consisting of: Hematoxyline and Eosin (H&E) and anon-labelled scan.
 3. The computer implemented method of claim 1,wherein the non-labelled scan is selected from a group consisting of:raman spectroscopy, autofluorescence, darkfield, and pure contrast. 4.The computer implemented method of claim 1, wherein the permanent stainis selected from a group consisting of: a non-H&E stain, and a specialstain.
 5. The computer implemented method of claim 4, wherein thespecial stain is selected from a group comprising: Masson’s trichrome,Jones Silver H&E, and Periodic Acid Schiff (PAS).
 6. The computerimplemented method of claim 1, wherein the second image is created usingthe sample of tissue depicted in the first image after the first imageis captured, wherein the sample of tissue in the first image is treatedto remove the removable stain to create a cleared tissue sample, whereinthe permanent stain is applied to the cleared tissue sample to create apermanently stained sample, wherein the second image depicts thepermanently stained sample.
 7. The computer implemented method of claim6, further comprising: for the record: identifying a first plurality ofbiological features depicted in the first image, identifying a secondplurality of biological features depicted in the second image thatcorrespond to the identified first plurality of biological featuresdepicted in the first image; applying an optical flow process and/ornon-rigid registration process to the respective second image to computea respective aligned image, the optical flow process and/or non-rigidregistration process aligns pixels locations of the second image tocorresponding pixel locations of the first image using optical flowcomputed between the second plurality of biological features and thefirst plurality of biological features; wherein the ground truthindicated by the second image comprises the aligned image.
 8. Thecomputer implemented method of claim 1, further comprising: creating aplurality of imaging multi-record training datasets, each imagingmulti-record training dataset comprising a different permanent staindepicted in the ground truth indicated by the second image, and traininga plurality of virtual stainer machine learning models on the pluralityof imaging multi-record training datasets.
 9. The computer implementedmethod of claim 1, wherein the virtual stainer machine learning modelcomprises a generative adversarial network (GAN) comprising a generatornetwork and discriminator network, wherein training comprises trainingthe generative network for generating the virtual image using a lossfunction computed based on differences between pixels of the secondimage of a record and pixels of an outcome of a virtual image created bythe generative network in response to an input of the first image of therecord, and according to an ability of the discriminator network todifferentiate between the outcome of the virtual image created by thegenerative network and the second image.
 10. A computer implementedmethod for generating an image of virtually stained tissue, comprising:feeding a target image of a sample of tissue of a subject stained with aremovable stain into a virtual stainer machine learning model trainedaccording to claim 1; and obtaining a synthetic image of the sampledepicting the sample of tissue stained with a permanent stain.
 11. Thecomputer implemented method of claim 10, further comprising selecting atleast one virtual stainer machine learning model from a plurality ofvirtual stainer machine learning models each trained on a differentimaging multi-record training dataset comprising a different permanentstains depicted in the ground truth indicated by the second image,wherein the target image is fed into each selected virtual stainermachine learning model.
 12. The computer implemented method of claim 10,further comprising analyzing the target image to identify a type of thesample of tissue from a plurality of types of sample tissue, andselecting the at least one virtual stainer machine learning modelaccording to the type of the sample, wherein the plurality of virtualstainer machine learning models are trained on different imagingmulti-record training dataset comprising different permanent stains fordifferent types of tissues.
 13. A computer implemented method fortraining a virtual stainer machine learning model, comprising: creatingan imaging multi-record training dataset, wherein a record comprises: afirst image of a sample of tissue of a subject stained with a permanentstain, a ground truth indicated by a second image of the sample oftissue of the subject stained with a removable stain; and training avirtual stainer machine learning model on the imaging multi-recordtraining dataset for generating a virtual image depicting the removablestain in response to an input image depicting the permanent stain. 14.The computer implemented method of claim 13, wherein the sample oftissue is first stained with the removable stain to create a removablestain stained sample, the second image of the removable stain stainedsample is captured prior to capture of the first image, the removablestain stained sample is treated to clear the removable stain to create acleared tissue sample, the permanent stain is applied to the clearedtissue sample to create a permanent stain stained sample, wherein thefirst image depicting the permanent stain stained sample is captured.15. A computer implemented method for generating an image of virtuallystained tissue, comprising: feeding a target image of a sample of tissueof a subject stained with a permanent stain into a virtual stainermachine learning model trained according to claim 12; and obtaining asynthetic image of the sample depicting the sample of tissue stainedwith a H&E stain.
 16. The computer implemented method of claim 15,wherein the permanent stain comprises an immunohistochemistry (IHC)marker stain.
 17. The computer implemented method of claim 15, whereinthe permanent stain is selected from a group comprising PDL1, HER2immunohistochemistry (IHC) stain.
 18. A computer implemented method forgenerating an image of virtually stained tissue, comprising: feeding atarget image of a sample of tissue of a subject stained with a removablestain into a virtual stainer machine learning model running on acomputer; and generating a synthetic image of the sample depicting thesample of tissue stained with a permanent stain as an outcome of thevirtual stainer machine learning model.
 19. A computing devicecomprising at least one processor executing a code for at least one oftraining a virtual stainer machine learning model and performinginference by the virtual stainer machine learning model, according tothe method of claim
 1. 20. A non-transitory medium storing programinstructions for at least one of training a virtual stainer machinelearning model and performing inference by the virtual stainer machinelearning model, which when executed by at least one processor, cause theat least one processor to perform features according to the method ofclaim 1.